Blobs:BlobMerger

Cancelled Posted Jul 22, 2010 Paid on delivery
Cancelled Paid on delivery

We need a Linux comandline program, which combines several S3 objects into one.

## Deliverables

# BlobMerger

## Technologies used

It is a Linux shell application, which works with TCP sockets and HTTP protocol.

Application has just one thread. **Multiplexing** socket handle mode is expected.

## Description

Application receives merge tasks through STDIN. Task will contain *signed* urls on the file parts, and *signed* POST request parameters, required to create new S3 file.

File combining should be done **without** file buffer in the local filesystem.

In-memory buffer should be used, with size of 4 Kb or more. The exact buffer size to use is set in the config file.

Combining is a simultaneous reading one file part and writing the result file.

## Specification

Application reads data from STDIN in the following format:

**ACTION <session_id|QUIT>** (message **ACTION QUIT** shuts down the application)

**DESTURL**

**DESTPARAMCOUNT**

(sequence of exact *DESTPARAMCOUNT* params)

**DESTPRAM <name=value>**

**DESTPRAM <name=value>**

**DESTPRAM <name=value>**

**…**

**SOURCECOUNT**

(sequence of exact *SOURCECOUNT* SOURCE entities)

**SOURCE <????? ? s3>**

**SOURCE <????? ? s3>**

**SOURCE <????? ? s3>**

**…**

Once all message is recieved, application writes to the STDOUT message

**WAIT**

Message handling is started.

Once task is completed, application writes to the STDOUT message

**DONE **

After that application is waiting for the next task.

Or

**ERROR** ??" if error happened and application is going to exit with error. Error description should contain all required information and be human-readable.

## Implementation

* Application should use select() with two filehandles : GETHANDLE and PUTHANDLE ??" one for reading, another one for writing.

* Application should write into the logfile all commands, that were recieved on STDIN and wrote on STDOUT, replacing value of Signature with asterixes everywhere.

* In case of connection for read file part is failed, application should retry three more times, if still no success, then application should exit with error.

* In case od connection for write result file is failed, application should exit with error.

* While interactiong with S3, application should check HTTP Statuses of the S3 responses and exit with error, once status is unexpected.

* Exiting with error: Application should write error to the logfile first. If the last command written to STDOUT was WAIT, than error message should be written to the STDOUT.

* Log file entries should have the following format:

**YYYY-MM-DD HH:MM PID sessionId message**

* Application should write two log files simultaneously: [login to view URL] and blob_merger_session_id.log.

If application finishes its work on current session successfully and there were no errors during the session, than [login to view URL] should be deleted.

* Application should read config text file, where buffer size and log files directory path are defined. Config file should be at ~/.blob_merger

C Programming Linux PHP Script Install Shell Script Software Architecture Software Testing UNIX

Project ID: #3591864

About the project

Remote project Active Aug 1, 2010