SFTP to Google Cloud Storage (GCS)
/ Google Cloud Storage to SFTP
Automated Data Transfers

Setting up your automated data transfer is very simple.

In this brief guide, we’ll walk you through creating and setting up an automated and scheduled data transfer between SFTP and Google Cloud Storage with Netice and your Google Cloud service account with the necessary permissions.

Start by logging in to Netice Data Transfer Platform

Note: If you don’t have an account yet, you can claim your free 30-day trial with no credit card or payment information required!

1. Create your transfer task

In main menu, click “Create a transfer task”.

Next, select your source and destination. In this case, either from SFTP to Google Cloud Storage or from Google Cloud Storage to SFTP.

2. Fill the task setup form

Note: Netice will transfer the newest matching data file based on your task setup inputs.

Now, the task setup form will open (the fields are mostly the same with SFTP to GCS transfers as they are with SFTP to BigQuery, as they both utilize Google Cloud Storage):

Below are the descriptions and instructions for every field in SFTP to Google Cloud Storage transfer automation:

SFTP Host

Description

This field is for entering the hostname or IP address of your SFTP server.

Instruction

Input your SFTP server address, for example, sftpserver@example.com.

SFTP Username

Description

The username for authenticating with your SFTP server.

Instruction

Enter your SFTP username, e.g., example.

SFTP Port

Description

The port number for your SFTP connection, usually 22.

Instruction

Input 22 unless your SFTP server uses a different port.

SFTP Private Key (PEM Format RSA Private Key)

Description

Your PEM format RSA private key for secure authentication.

Instruction

If your SFTP server uses key authentication, paste your private key content here. Otherwise leave empty. If your server is also passphrase-protected, you can provide the passphrase in the Advanced Options. If you don’t know the key, please contact your organization’s SFTP administrator. For the extensive explanations on security measures which we take to secure your secrets, please refer to Netice’s Security Policy.

SFTP Password

Description

The password for your SFTP account. An alternative way to authenticate to SFTP servers, substituting a private key.

Instruction

Enter your password if your SFTP server uses this authentication method. Otherwise leave empty. For the extensive explanations on security measures which we take to secure your secrets, please refer to Netice’s Security Policy.

SFTP Path to Directory

Description

The path to the directory on your SFTP server for which to transfer files to/from.

Instruction

Input / for the root directory or /folder1/folder2/ for specific folders.

Google Cloud Storage Bucket Name

Description

The name of the Google Cloud Storage bucket where data will be transferred. Google Cloud Storage acts as the middle point in Netice’s transfers between SFTP and Google BigQuery.

Instruction

Enter the bucket name, e.g., netice_transfer_9999999. The bucket will be created if it doesn’t already exist.

Days Ago

Description

Defines how many days back to retrieve data from your source. If a file or data is marked as older than your input, it won’t be considered for transfers.

Instruction

Set to 1 to retrieve the latest file. This value specifies how many days back to look for files.

Schedule Type

Description

The type of scheduled increments for transferring data. Possible values depending on your subscription type are Hourly, Daily, Weekly & Monthly.

Instruction

Select e.g. Daily to schedule transfers in daily increments.

Interval

Description

The numeric interval in what you chose as Schedule Type between each automated transfer.

Instruction

Example: if you have selected Schedule Type as daily, enter 1 to run the transfer daily or 3 to run the transfer every 3 days.

Start Date and Time

Description

The start date and time for the scheduled transfer.

Instruction

Input the date and time in mm/dd/yyyy --:-- -- format. The future runs of your transfer task will be set relative to the Start Date and Time you have selected. This means that if you have, for example, selected 1/1/2025 02:00 as the starting date and time, the future runs will take place on 1/2/2025 02:00, 1/3/2025 02:00 and so on. In addition to the scheduled future transfer, you can set a one-time immediate transfer of data when you submit the task by selecting “Yes” in the Test run field.

End Date and Time (Optional)

Description

Optional field to specify when to end the scheduled transfers, ie. when to not do the scheduled transfer anymore. This will not delete the task, although it will pause it. A task ending will not have any effect on your subscription.

Instruction

Input the end date and time in mm/dd/yyyy --:-- -- format if needed. If you don’t want to specify an end date, please leave it untouched.

Time Zone

Description

The time zone for the scheduled transfer timings. By default set to your device’s time zone.

Instruction

If the default time zone selection is incorrect, please select your time zone manually from the drop-down list, e.g., Europe/Helsinki.

CSV Field Delimiter

Description

The delimiter character used in your CSV files. By default set to comma.

Instruction

Input , for comma or specify another character such as ; if your CSV uses a different delimiter.

Google Cloud Platform Service Account Key

Description

Your service account key for authenticating Google Cloud operations in products such as BigQuery and Cloud Storage.

Instruction

Paste the full JSON content of your service account key here. The information is encrypted for security and retained securely for scheduled transfer use. For further instructions, please refer to Netice’s GCP Service Account Guide. For the extensive explanations on security measures which we take to secure your secret keys, please refer to Netice’s Security Policy.

SFTP Passphrase

Description

Passphrase for the SFTP private key, if it is passphrase-protected.

Instruction

Leave empty if your private key is not passphrase-protected. Otherwise, provide the passphrase. For the extensive explanations on security measures which we take to secure your secrets, please refer to Netice’s Security Policy.

File Name Contains

Description

Filter to specify which files to transfer based on their names.

Instruction

Leave empty unless you need to specify a pattern to match multiple files.

Empty the SFTP Folder After Successful Transfer

Description

Option to delete files from the SFTP folder after transfer. (From SFTP to BigQuery).

Instruction

Select No to retain files on the server after transfer, or yes to have them deleted.

Next 5 Schedules

Description

Shows the next five scheduled transfer times to provide the user a clear and practical illustration of when the schedules take place.

Instruction

No input needed; this is for informational purposes.

Do a Test Run Now?

Description

Option to perform a test run of the data transfer.

Instruction

Select Yes to perform a test run. For the test run, you need to have the file to be transferred in correct place and as required by your inputs and selections in the task setup form. If the test transfer is successful, the task will be scheduled. If the transfer is not successful, you will be prompted of the error message and the task will not be scheduled yet. If you choose to not do the test transfer, the transfer will be normally scheduled to start in the start time as it would have been with a successful test run.

3. Submit the task

Please wait patiently while the task is being submitted, especially if a test run is scheduled as it may take some minutes to finish the test transfer.

After submitting the task, you will be returned to your tasks.

If your task submission had been successful, you will see a one-time message on green saying “Task processed successfully!”. If the task submission was not successful (usually because of a failed test run), you will be prompted of the error message on red.

Please note that the mentioned one-time messages are different from the scheduled run messages. Below is an example of what a task screen with errors and successful runs for separate tasks may look like with past scheduled runs:

Once you have successfully submitted your task, you will see it appear in the Tasks page as seen below, and the “Last scheduled run status” will appear there once it has run at least once according to its schedule.

Further questions?

For any further questions please refer to our Documentation or contact our Support. We are here to help you to be successful!

To read more about Google Cloud Platform service accounts, please refer to Netice’s service account guide. For further information on what service accounts are and what they’re used for, please refer to this official article by Google.