File Mover Blog

  • 09 May

    I want to save the files to a sent folder after uploading to FTP

    A: I want to save the files to a sent folder after uploading to FTP?

    We look for files from a Windows Source and then send them to a FTP site. Destination memory is enabled so files are only ever sent once.  I want to save the sent files in Windows SENT folder as once they go to the customer FTP site they move or delete the files so we also store them on our side.

    Q: This is possible. You’ll need to add two destinations:

    • Destination 1 would be the FTP server (Important, must be the first Destination)
    • Destination 2 would be the Windows SENT folder

    limagito file mover destination setup

    In the Function Setup, please enable ‘Destination Memory  & Exit cylcus on Error’. In this case only when FTP succeeds the file will go to the Windows SENT folder. If it doesn’t succeed we’ll try to send the file to the FTP destination again during the next scan.

    limagito file mover function setup

    #filetransfer #ftp #mft

    If you need any info  about this request, please let us know.

    Best regards,

    Limagito Team

    By Limagito-Team Backup FTP ,
  • 07 May

    Copy only the modified contents of a file

    Q: We are using Limagito for different projects and we have a query to its functionality. We want to see that it can copy only the modified contents of a file  in a new file with same name. Below is an example on what is being appended and how.

    Let us use the following example: Equipment_AE_Alarm_Event0_2023-05-04_00.00.01.csv.

    • The original content of this csv file:

    “Time;””AlarTxt””;””AlmPrio””;””Almusr””;””acksts”””
    “2023-05-03 00:00:02;””sample text1″”;200;KLMN;1″
    “2023-05-03 00:00:03;””sample text2″”;300;KLMN;1″
    “2023-05-03 00:00:04;””sample text3″”;100;KLMN;0″
    “2023-05-03 00:00:05;””sample text4″”;200;KLMN;1″
    “2023-05-03 00:00:06;””sample text5″”;200;KLMN;1″

    • Expected content of the output file would be the same when we scan this file for the first time:

    “Time;””AlarTxt””;””AlmPrio””;””Almusr””;””acksts”””
    “2023-05-03 00:00:02;””sample text1″”;200;KLMN;1″
    “2023-05-03 00:00:03;””sample text2″”;300;KLMN;1″
    “2023-05-03 00:00:04;””sample text3″”;100;KLMN;0″
    “2023-05-03 00:00:05;””sample text4″”;200;KLMN;1″
    “2023-05-03 00:00:06;””sample text5″”;200;KLMN;1″

    • Data is being added to this csv so the content of file during the next scan is showing more entries:

    “Time;””AlarTxt””;””AlmPrio””;””Almusr””;””acksts”””
    “2023-05-03 00:00:02;””sample text1″”;200;KLMN;1″
    “2023-05-03 00:00:03;””sample text2″”;300;KLMN;1″
    “2023-05-03 00:00:04;””sample text3″”;100;KLMN;0″
    “2023-05-03 00:00:05;””sample text4″”;200;KLMN;1″
    “2023-05-03 00:00:06;””sample text5″”;200;KLMN;1″
    “2023-05-03 00:00:06;””sample text6″”;200;KLMN;0″
    “2023-05-03 00:00:06;””sample text7″”;200;KLMN;1″
    “2023-05-03 00:00:08;””sample text8″”;200;KLMN;0″
    “2023-05-03 00:00:10;””sample text9″”;200;KLMN;1″
    “2023-05-03 00:00:12;””sample text10″”;200;KLMN;0″

    • Expected content of the output file after the second scan should be:

    “Time;””AlarTxt””;””AlmPrio””;””Almusr””;””acksts”””
    “2023-05-03 00:00:06;””sample text6″”;200;KLMN;0″
    “2023-05-03 00:00:06;””sample text7″”;200;KLMN;1″
    “2023-05-03 00:00:08;””sample text8″”;200;KLMN;0″
    “2023-05-03 00:00:10;””sample text9″”;200;KLMN;1″
    “2023-05-03 00:00:12;””sample text10″”;200;KLMN;0″

     

    A: This should be possible using our scripting option. We created a script that will keep the data of the csv files it already handled in a separate folder (= memory folder). These memory files will be used during next scans to only create output files where the content has new data.

    For the test, we added 3 folders:

    • Input
    • Memory
    • Output

    Be sure the ‘Memory’ folder is a local folder on the system where our File Mover is running.

    Modified contents of a file example

    Our Source will be the Input folder:

    Limagito file mover using WIN as source

    We added ‘*.csv’ as Filename Include filter because we only want to scan for csv files:

    limagito file mover filename filter

    Optional: we changed the ‘Source File Sort Order’ option to ‘File last write date ascending (Old to New)’. This will handle the oldest csv files first.

    limagito file mover scan older files first

    The function is set to ‘Copy Files’:

    limagito file mover function setup

    As Destination we’ll use our ‘Pascal Script’ option:

    You’ll need to adjust the following 2 constant values ( always end the constant with a \ ):

    • ctMemPath ( = path were we’ll keep our memory data )
    • ctOutPath ( = output path)

    We added a timestamp to the output filename. The format we used is ‘YYYYMMDDHHNNSS’. The reason we did this is to prevent data loss (in case the next system would not pickup the output files).

     

    Var
      tmpEntry, tmpText, tmpFileIn, tmpFileMem, tmpFileOut, tmpFileExtIn, tmpFileExtOut: String;
      tmpList, tmpListMem: TStringList;
    Const
      ctMemPath = 'C:\Test\Astra\Memory\';
      ctOutPath = 'C:\Test\Astra\Output\';
    Begin
      psExitCode:= 0;
      tmpFileIn := psFilePath + psFileName;
      tmpFileMem := ctMemPath + psFileName;
      tmpFileExtIn := ExtractFileExt(psFileName);
      tmpFileExtOut := '.' + FormatDateTime('YYYYMMDDHHNNSS', Now) + tmpFileExtIn;
      tmpFileOut := psStringReplace(psFileName, tmpFileExtIn, tmpFileExtOut);
      tmpFileOut := ctOutPath + tmpFileOut; 
      // Create Var
      tmpList := TStringList.Create; tmpListMem := TStringList.Create;
      Try
        Try 
          tmpList.LoadFromFile(tmpFileIn);
          // Check if file is already available in Memory folder
          If tmpList.Count > 1 Then
          Begin
            tmpEntry := tmpList.Strings[0]; tmpText := tmpList.Text; 
            If FileExists(tmpFileMem) Then
            Begin
              psLogWrite(1, '', psFileName + ' available in memory folder: ' + ctMemPath);                 
              Try
                tmpListMem.LoadFromFile(tmpFileMem);                          
                tmpList.Text := psStringReplace(tmpText, tmpListMem.Text, '');
                // tmpList.Insert(0, tmpEntry);
                psExitCode := 1;         
              Except
                psLogWrite(1, '', 'Error loading data from: ' + tmpFileMem);
              End;          
            End Else
            Begin
              psExitCode := 1;
              psLogWrite(1, '', psFileName + ' not available in memory folder: ' + ctMemPath);               
            End;  
            If tmpList.Count > 1 Then
            Begin  
              // Save to output folder
              If psExitCode = 1 Then 
              Begin
                psExitCode := 0;
                Try
                  psLogWrite(1, '', 'Saving data to: ' + tmpFileOut);       
                  tmpList.SaveToFile(tmpFileOut);
                  psExitCode := 1;
                Except
                  psLogWrite(1, '', 'Error saving data to: ' + tmpFileOut);
                End;  
              End;
              // Save to Memory folder      
              If psExitCode = 1 Then
              Begin
                psExitCode := 0;
                Try
                  psLogWrite(1, '', 'Saving data to: ' + tmpFileMem);
                  tmpList.Text := tmpText; tmpList.Delete(0);       
                  tmpList.SaveToFile(tmpFileMem);
                  psExitCode := 1;
                Except
                  psLogWrite(1, '', 'Error saving data to: ' + tmpFileOut);
                End;          
              End;
            End;        
          End Else
            psLogWrite(1, '', 'No data available in: ' + tmpFileIn);   
        Except
          psLogWrite(1, '', 'Error loading data from: ' + tmpFileIn);
        End;
      Finally
        tmpList.free; tmpListMem.Free;
      End;
    End.
    

    limagito file mover pascal script as destination

    This blog article is just a basic example of the customers request and can be tweaked so that it meets your requirements. Just contact us if you need help.

    Don’t forget to add an extra Rule which will cleanup the Memory folder on regular basis.

    #FileTransfer #csv #scada #MFT

    If you need any info  about this ‘Copy only the modified contents of a file’ request, please let us know.

    Best regards,Limagito Team

  • 27 Apr

    Centralized log file in industry standard CSV format

    Q: Could a centralized log file in industry standard CSV format be implemented within the file mover software? This would be a log file that *all* jobs could be configured to log to. We have looked at the individual History log files and even though we could probably combine them into a single, daily file to push up to Snowflake – it appears they are not character delimited and only have a space between columns which would be very difficult if not impossible to parse out into individual fields.

    A: We added a global logging option in version v2023.2.26.0.

    Feedback customer:

    We anticipate wanting to ship our log files up to a remote system every 5 or 10 minutes. Since the log file is locked we can’t force this to happen and of course it would not be wise for us to mess with the “active” log file anyway. Instead of (or in addition to) the “Start new log every day”, could there be a “Start new log every __ minutes” and let us adjust the number of minutes before a log file rolls over to a new one?

    Here is the custom message format I was testing with: %{DATETIME}|%{LEVEL}|%{LEVELINT}|%{HOSTNAME}|%{RULEID}|%{MESSAGE}

    It appears the RuleId is always 0, instead of pulling the actual RuleId value
    2023-04-24 15:05:58|INFO|1|CHHMOVER01|0|cerner – XR HIM2 [SFTP] CheckSFTPConnected, SB Request Path after Login: /

    It appears the RuleName is included as a prefix to the message:

    2023-04-24 15:05:58|INFO|1|CHHMOVER01|0|cerner – XR HIM2 [SFTP] CheckSFTPConnected, SB Request Path after Login: /

    Preferably this would be a separate tag (ex. %{RULENAME}) and not a prefix on the message so we could control the output better, for example:

    %{DATETIME}|%{LEVEL}|%{LEVELINT}|%{HOSTNAME}|%{RULEID}|%{RULENAME}|%{MESSAGE} would output a message like this:

    2023-04-24 15:05:58|INFO|1|CHHMOVER01|0|cerner – XR HIM2|[SFTP] CheckSFTPConnected, SB Request Path after Login: /

    A: We added some new options and parameters in version v2023.4.27.0

    Limagito file mover global logging

    We added options to remove certain information like “Rule Name” from the default log message. We enabled “Use Custom Message Format” and added the format from the customer as test.

    limagito file mover Centralized log file

    The following predefined variables can be used to produce a custom message output format:

    • DATETIME : Date & time log item occurs
    • DATE : Date log item occurs
    • TIME : Time log item occurs
    • LEVEL : Level or Eventype
    • LEVELINT : Level as numeric
    • MESSAGE : Message sent to logger
    • ENVIRONMENT : Customizable variable (normally Production, Test, etc)
    • PLATFORM : Customizable variable (normally Desktop, Mobile, etc)
    • APPNAME : Customizable variable (by default set as filename without extension)
    • APPVERSION : Application file version
    • APPPATH : Application run path
    • HOSTNAME : Computer name
    • USERNAME : Logged user name
    • OSVERSION : OS version
    • CPUCORES : Number of CPU cores
    • THREADID : Thread ID log item set
    • RULEID : RuleID number
    • RULENAME : RuleName
    • LOGSOURCE : Description of log source

    Logging result:

    limagito file mover runtime log

    TODO:

    We anticipate wanting to ship our log files up to a remote system every 5 or 10 minutes. Since the log file is locked we can’t force this to happen and of course it would not be wise for us to mess with the “active” log file anyway. Instead of (or in addition to) the “Start new log every day”, could there be a “Start new log every __ minutes” and let us adjust the number of minutes before a log file rolls over to a new one?

    #FileTransfer #Logging #MFT

    If you need any info  about this ‘Centralized log file’, please let us know.

    Best regards,

    Limagito Team

    By Limagito-Team Logging ,
1 24 25 26 27 28 29 30 134
SEARCH