Close

Azure filesize limitations

While you, as a Microsoft developer, are inclined to approach Azure from the opportunities side, it’s also worth looking at the limitations. I didn’t find a web source with a full list of limitations. That’s why I simply start by discussing two file size limitations here:

FTP:

The FTP connector trigger supports max 50 MB. I found a reaction that states: The FTP trigger has a size limit, but FTP does support upload and download of files upto 1 GB via other actions. You should be able to use metadata-only trigger followed by Get File Content action to retrieve the action data (Feb 2018). I’m not sure if that’s true. At a current project we use a recurrence trigger and a FTP action List Files in Folder. When looping through the files it’s actually checked whether file size is less than 50MB. Only then, the file content is retrieved via FTP action Get File Content.

Question for the next occasion when I encounter this issue: is there a size limit on FTP action Get File Content? When you actually do have a size limit, you’ll have to split up large files before processing by a logic app. You could do this in an Azure function. Note that NuGet package SSH.NET is used for the SFTPClient. Also note that a parameter Bytes_per_File (=30MB) is passed to the function. The function actually reads the file from FTP, splits it up, stores the files in blob storage and returns the filenames in a string collection. The Logic App processes the string collection with the filenames (incl. blobpath) and puts a message with the blobpath in a queue for further processing. Example:

public static async Task<string> Handle([ActivityTrigger] RequestData data, TraceWriter log)
{

// CopyArguments contains (in Json format):
// “bytes_per_file”, “destination_folder”, “password”, “path”, “server”, “source_guid”, “treat_first_line_as_header”, “username”
var args = new CopyArguments();

args.FromJObject(JObject.Parse(data.Json));

var recordsTotal = 0;
var files = new List<string>();

var minLines = args.TreatFirstLineAsHeader ? 2 : 1;
var blobClient = StorageAccount.CreateCloudBlobClient();
var connectionInfo = new ConnectionInfo(args.Server, args.Username, new PasswordAuthenticationMethod(args.Username, args.Password), new PrivateKeyAuthenticationMethod(“rsa.key”));

using (var sftp = new SftpClient(connectionInfo))
{
sftp.Connect();

using (var stream = sftp.OpenRead(args.SourcePath))
{
using (var reader = new StreamReader(stream))
{
string header = null;
ulong headerSize = 0;

if (reader.Peek() >= 0) // there is data …
{
var sw = new Stopwatch();

sw.Start();

var destinationPath = args.GetDestinationPath(files.Count + 1);
var outStream = await GetBlobStream(blobClient, destinationPath, logging);
var writer = new StreamWriter(outStream);
ulong fileSize = 0;
var recordsFile = 0;

while (reader.Peek() >= 0)
{
var line = reader.ReadLine();

if (string.IsNullOrEmpty(line))
continue;

var lineSize = Convert.ToUInt64(Encoding.UTF8.GetByteCount(line));

if (header == null && recordsTotal == 0 && args.TreatFirstLineAsHeader)
{
header = line; // backup the header …
headerSize = lineSize;
}
else
{
recordsTotal++;
recordsFile++;
}

if (recordsFile >= minLines && args.BytesPerFile > 0 && fileSize + lineSize > args.BytesPerFile)
{
writer.Flush();
writer.Dispose();
outStream.Dispose();

files.Add($”/{string.Join(“/”, destinationPath)}”);

sw.Stop();

recordsFile = 0;
fileSize = 0;

sw = new Stopwatch();
sw.Start();

destinationPath = args.GetDestinationPath(files.Count + 1);
outStream = await GetBlobStream(blobClient, destinationPath, logging);
writer = new StreamWriter(outStream);

if (args.TreatFirstLineAsHeader)
{
await writer.WriteLineAsync(header);
fileSize = headerSize;
}
}

await writer.WriteLineAsync(line);

fileSize += lineSize;
}

writer.Flush();
writer.Dispose();
outStream.Dispose();

files.Add($”/{string.Join(“/”, destinationPath)}”);

sw.Stop();

}
}
}

return FunctionResponse.CreateExportSuccess(recordsTotal, files.ToArray()).ToString();
}
}

private static Task<CloudBlobStream> GetBlobStream(CloudBlobClient client, string[] path, Logging logging)
{
var containerRef = client.GetContainerReference(path[0]);
var storagePath = string.Join(“/”, path.Skip(1));
var blobRef = containerRef.GetBlockBlobReference(storagePath);

blobRef.Properties.ContentType = “text/csv”;

logging.Info($”Opened write stream to {storagePath}”);

return blobRef.OpenWriteAsync();
}
}

public class RequestData
{
public string Json { get; set; }
}
}

File:

You can’t get content of a file from a fileshare when it exceeds 30MB. I don’t know if it makes a difference, but I didn’t use Azure file storage, but an externally accessible fileshare. This is the http 413 code (Aug 2018) I received:

{
“status”: 413,
“message”: “The file contains 129.521 megabytes which exceeds the maximum 30 megabytes.\r\nclientRequestId: fc9f147c-444b-45d5-8bd6-6aead2b5ba03”,
“source”: “filesystem-we.azconn-we.p.azurewebsites.net”
}

In this case we only wanted to a file transfer, no file conversion. So, the solution was to zip the file before sending, which reduced the file size of the .dat file to 5MB. Problem solved.

Leave a Reply

Your email address will not be published. Required fields are marked *