fixpp stands for FIX by C++ PreProcessor

Motivation

The idea is to use CPP preprocessing instructions along with standard Linux CLI tools(bash,sed,grep) to generate FIX API for a given venue.

Mindset:

• No third party dependencies. All the generated code is yours and can go straight into your lib or app.
• Repeating groups are supported.
• You can strip off all useless standard FIX tags to fit your venue specs.
• FIX tags and enums are treated as integers and not strings thus removing transformations.

Receiving (read more):

• Lightweight and low latency FIX parsing with reusable memory and objects.
• Flexible formatting with available predefined TTY color styles.
• Pretty printing with GDB and VSCode.

Sending (read more):

• Low latency FIX message building with reusable memory and objects.
• For a given message type, only fields changing between two sends are to be updated.

You have to know:

• fixpp is not a FIX engine (yet)
• it runs in an optimistic mode and relies on the venue FIX conformance (this saves quite a few CPU cycles...)

Performance

Tags and enums are not converted. For instance the sequence like "|49=" will be inserted as a single integer assignment with value 0x3d'39'34'01. On reception, the string "49" will not be translated to decimal value 49 but rather reinterpreted as 0x39'34.

Results obtained in a tight loop on i9-9900K @ 5GHz

action	message type	length	time, ns	CPU cycles	HW instructions
decode	ExecReport	170	75	375	911
decode	MarketDataSnapshotFullRefresh*	330	122	608	1647
encode	NewOrderSingle	170	68	338	909
update	NewOrderSingle	170	61	303	821

* 6 repeating groups

Screenshots

TTY output

Debugger view in vscode

Input

Based on your venue FIX API prepare files Fields.def, Groups.def and Messages.def. One can pick up ready to use files from examples/spec and adjust Groups.def and Messages.def. Fields.def will be cleaned automatically during generation. If you have already a Quickfix XML file you can use the xml2cpp converter to generate the definitions.

src/spec/Fields.def

FIX_FIELD_BEGIN_STRING( FIX.4.4 )

FIX_FIELD_DECL( Account                       ,      1, STRING         )
FIX_FIELD_DECL( AvgPx                         ,      6, PRICE          )
FIX_FIELD_DECL( BeginSeqNo                    ,      7, SEQNUM         )
FIX_FIELD_DECL( BeginString                   ,      8, STRING         )
FIX_FIELD_DECL( BodyLength                    ,      9, LENGTH         )
FIX_FIELD_DECL( CheckSum                      ,     10, STRING         )
FIX_FIELD_DECL( ClOrdID                       ,     11, STRING         )
...
FIX_ENUM_BEGIN( MsgType )
FIX_ENUM_DECL( MsgType, HEARTBEAT                        , SOHSTR(0) )
FIX_ENUM_DECL( MsgType, TEST_REQUEST                     , SOHSTR(1) )
FIX_ENUM_DECL( MsgType, RESEND_REQUEST                   , SOHSTR(2) )
FIX_ENUM_DECL( MsgType, REJECT                           , SOHSTR(3) )
FIX_ENUM_DECL( MsgType, SEQUENCE_RESET                   , SOHSTR(4) )
FIX_ENUM_DECL( MsgType, LOGOUT                           , SOHSTR(5) )
FIX_ENUM_DECL( MsgType, EXECUTION_REPORT                 , SOHSTR(8) )
FIX_ENUM_DECL( MsgType, LOGON                            , SOHSTR(A) )
FIX_ENUM_DECL( MsgType, ORDER_SINGLE                     , SOHSTR(D) )
FIX_ENUM_DECL( MsgType, MARKET_DATA_REQUEST              , SOHSTR(V) )
FIX_ENUM_DECL( MsgType, MARKET_DATA_SNAPSHOT_FULL_REFRESH, SOHSTR(W) )
FIX_ENUM_DECL( MsgType, MARKET_DATA_INCREMENTAL_REFRESH  , SOHSTR(X) )
FIX_ENUM_DECL( MsgType, MARKET_DATA_REQUEST_REJECT       , SOHSTR(Y) )
FIX_ENUM_END
...

src/spec/Groups.def

FIX_MSG_GROUP_BEGIN( MDEntries, MDEntryType )
FIX_MSG_FIELD( MDEntryPx )
FIX_MSG_FIELD( Currency )
FIX_MSG_FIELD( MDEntryPositionNo )
FIX_MSG_FIELD( MDEntrySize )
FIX_MSG_FIELD( MDEntryDate )
FIX_MSG_FIELD( MDEntryTime )
FIX_MSG_GROUP_END
...

src/spec/Messages.def

FIX_MSG_BEGIN( Header )
FIX_MSG_FIELD( BeginString )
FIX_MSG_FIELD( BodyLength )
FIX_MSG_FIELD( MsgType )
FIX_MSG_FIELD( SenderCompID )
FIX_MSG_FIELD( TargetCompID )
FIX_MSG_FIELD( MsgSeqNum )
FIX_MSG_FIELD( SendingTime )
FIX_MSG_END

FIX_MSG_BEGIN( Heartbeat )
FIX_MSG_FIELD( TestReqID )
FIX_MSG_END

FIX_MSG_BEGIN( Logon )
FIX_MSG_FIELD( EncryptMethod )
FIX_MSG_FIELD( HeartBtInt )
FIX_MSG_FIELD( ResetSeqNumFlag )
FIX_MSG_FIELD( Username )
FIX_MSG_FIELD( Password )
FIX_MSG_END

FIX_MSG_BEGIN( Logout )
FIX_MSG_FIELD( Text )
FIX_MSG_END
...

See more in examples.

Output

/path/to/fixpp/generate.sh -d MYPRJ/src/myprj/fix -s MYPRJ/src/spec -i myprj/fix -n venue::fix -p MYPRJ/src/gdb

Where the options are:

• -d destination directory
• -s specification directory with .def files
• -i prefix to use in include statements (ex: #include <prefix/Field.h>)
• -n namespace to use
• -p pretty printers destination dir

It will create the following tree:

MYPRJ
├── Makefile
|
└── src
    |
    ├── spec // your input
    |   ├── Fields.def
    |   ├── Groups.def
    |   └── Messages.def
    │
    ├── gdb // generated files
    │   └── printers.py
    |
    └── myprj
        │
        └── fix // generated files
            ├── Fields.cpp
            ├── Fields.h
            ├── FixApi.h
            ├── Groups.cpp
            ├── Groups.h
            ├── Messages.cpp
            ├── Messages.h
            └── SenderApi.h

A fully generated and committed primitive project is available in examples/order.

Receiving API

Accessing message fields

class BusinessLogic
{
    void onMessage( const MessageExecutionReport & msg )
    {
        // for mandatory fields it is safe to get the value:
        Quantity qty = msg.getOrderQty();

        // for optional fields:
        double px = msg.isSetPrice() ? msg.getPrice() : 0;

        ...
    }

    void onMessage( const MessageMarketDataSnapshotFullRefresh & msg )
    {
        ...
    }
};

Scanning

In most cases a latency sensitive implementation will tend to reuse objects and avoid allocations on the critical path. Here we suggest keeping a Header object and messages separately. The parsing is carried out in two steps:

• scanning the header first,
• then, after identifying the message type, scanning the relevant message body.

    // my class attribute:
    Header                               _header;
    MessageExecutionReport               _execReport;
    MessageMarketDataSnapshotFullRefresh _mdsfr;
    BusinessLogic                        _businessLogic;
    ...
    // after reading the message into fixString
    offset_t pos = _header.scan( fixString.data(), fixString.size() );
    switch( _header_.getRawMsgType() )
    {
        case MsgTypeRaw_EXECUTION_REPORT:
            _msgExecutionReport.scan( fixString.data() + pos, fixString.size() - pos );
            _businessLogic.onMessage( _execReport );
            break;

        case MsgTypeRaw_MARKET_DATA_SNAPSHOT_FULL_REFRESH:
            _mdsfr.scan( fixString.data() + pos, fixString.size() - pos );
            _businessLogic.onMessage( _mdsfr );
            break;
        ...
    }

Use ParserDispatcher

fixpp generates a class ParserDispatcher implementing the above switch. You have just to override onMessage(...) methods.

#include <tiny/Messages.h>
#include <cstring>

const char * buffer = "8=FIX.4.4" I "9=156" I "35=8" I ... I "10=075" I;

using namespace venue::fix;

class MyProcessor: public ParserDispatcher
{
    protected:

        virtual void unprocessedMessage( raw_enum_t msgType, MessageBase & msg ) override
        {
            auto it = MsgTypeEnums::itemByRaw.find( msgType );
            const char * typeName = it != MsgTypeEnums::itemByRaw.end() ? it->second->name : "unknown";
            std::cout << "unprocessed " << typeName << std::endl;
        }

        virtual void onMessage( MessageMarketDataSnapshotFullRefresh & msg ) override
        {
            std::cout << "processed " << getCurrentHeader().getMsgType() << " " << msg.getMessageName() << std::endl;
        }

};

int main( int args, const char ** argv )
{
    size_t len = strlen( buffer );
    const char * cursor = buffer;

    MyProcessor mp;
    while( cursor = mp.parseAndDipatch( cursor, len - ( cursor - buffer ) ) )
    {
    }

    return 0;
}

How to read from socket or file

    constexpr unsigned begStrAndBodyLenBytes = 20; // reasonably large initial number of bytes to read
    std::vector<char> recvbuffer( 4096 );
    while( source.read( &recvbuffer[0], begStrAndBodyLenBytes ) )
    {
        unsigned msgTypeOffset;
        len = parseMessageLength( &recvbuffer[0], msgTypeOffset ) + 7; // 7 = chsum length
        if( recvbuffer.size() < len + msgTypeOffset )
        {
            recvbuffer.insert( recvbuffer.end(), len + msgTypeOffset - recvbuffer.size() + 100 , 0 );
        }
        // read the remaining bytes
        if( source.read( & recvbuffer[begStrAndBodyLenBytes], len - begStrAndBodyLenBytes + msgTypeOffset ) )
        {
            mp.parseAndDipatch( &recvbuffer[0], len + msgTypeOffset );
        }
        else
        {
            break;
        }

    }

DIY

#include <myprj/fix/Messages.h>

const char * execReport = "8=FIX.4.4" I "9=332" I "35=8" I "49=foo" I "56=bar" I "52=20071123-05:30:00.000" I
"11=OID123456" I "150=E" I "39=A" I "55=XYZ" I "167=CS" I "54=1" I "38=15" I "40=2" I "44=15.001" I "58=EQUITYTESTING" I "59=0" I "32=0" I "31=0" I "151=15" I "14=0" I "6=0" I
"555=2" I "600=SYM1" I "624=0" I "687=10" I "683=1" I
                             "688=A" I "689=a" I
                             "564=1" I
                             "539=2" I "524=PARTY1" I "525=S" I
                                   "524=PARTY2" I "525=S" I
                                   "804=2" I "545=S1" I "805=1" I "545=S2" I "805=2" I
      "600=SYM2" I "624=1" I "687=20" I "683=2" I
                             "688=A" I "689=a" I
                             "688=B" I "689=b" I
"10=027" I;



    using namespace venue::fix;
    ...

    MessageHeader header;
    // pos = offset from message start
    offset_t pos = header.scan( execReport, strlen( execReport ) );

    MessageExecutionReport er;
    pos = er.scan( execReport + pos, strlen( execReport ) - pos );

    // print single field value
    std::cout << ' ' << FixOrdStatus << " = " << er.getOrdStatus() << std::endl;

    // print entire message
    std::cout << "\n\n -- Pretty Printing --" << std::endl;

    // use operator <<
    std::cout << fixstr( execReport, ttyRgbStyle ) << std::endl;

    // print flat and advance pos
    pos = 0;
    fixToHuman( execReport, pos, std::cout, ttyRgbStyle ) << std::endl;

    // indent groups
    pos = 0;
    fixToHuman( execReport, pos, std::cout, ttyRgbStyle, MessageExecutionReport::getFieldDepth ) << std::endl;

    // iterate over groups
    if( er.isSetNoLegs() )
    {
        unsigned noLegs = er.getNoLegs();
        for( unsigned legIdx = 0 ; legIdx < noLegs; ++legIdx )
        {
            const GroupLegs & leg = er.getGroupLegs( legIdx );
            std::cout << legIdx << ": side:" << leg.getLegSide() << " qty: " << leg.getLegQty() << std::endl;
        }
    }

Sending API

You will have to include SenderApi.h to build FIX messages with fixpp. It offers both

• low level buffer construction with FixBufferStream
• and reusable memory approach with ReusableMessageBuilder

FixBufferStream

This structure has two attributes: begin and end. Respectively pointing to the message's first and past last byte. Each time a new field is inserted end will shift forward accordingly. In most cases the fields will be appended as tag-value pairs:

execReport.append<ClOrdID>("OID4567");
execReport.append<QtyType>( QtyTypeEnums::UNITS.value );
execReport.append<Price>( 21123.04567, 2 );

It is also possible to push tags and values separately:

execReport.pushTag<ClOrdID>().pushValue("OID4567");

ReusableMessageBuilder

The idea behind is for a given FIX session:

• to reuse the header since most of it's fields will not change,
• to pre-compute the checksum for non-changing header's fields,
• to update only changing time within timestamps since the date does not change intra day.

This structure inherits the begin and end pointers from FixBufferStream. But begin refers to the first changing field like SendingTime for instance. The very first byte of the sending buffer will be pointed to by start. The latter will move each time the header's width changes. For example when the sequence number or body length change their widths.

buffer   start                msgType                                    sendingTime                  body
|        |                    |                                          |                            |
"..."   "8=FIX.4.4" I "9=315" I "35=W" I "49=foo" I "56=bar" I "34=1234" I "52=20190101-01:01:01.000" I "..."
                                                                         |                            |
                                                                         begin                        end

A typical scenario will be

    using namespace fix;
    using namespace fix::field;
    using namespace fix::message;
    ...

    /// before we send it

    // prepare
    ReusableMessageBuilder order( NewOrderSingle::getMessageType(), 512, 128 );
    order.header.append<SenderCompID>("ASENDER");
    order.header.append<TargetCompID>("ATARGET");
    order.header.pushTag<FieldMsgSeqNum>();
    order.header.finalize();

    // append SendingTime to the header
    auto constexpr tsLen  = TimestampKeeper::DATE_TIME_MILLIS_LENGTH;
    auto constexpr tsFrac = TimestampKeeper::Precision::MILLISECONDS;
    order.append<SendingTime>( TimestampKeeper::PLACE_HOLDER, tsLen );
    // initialize the timestamp keeper
    order.sendingTime.setup( order.end - tsLen, tsFrac );
    order.sendingTime.update();
    const unsigned sendingTimeLength = order.end - order.begin;
    ...
    /// sending it
    void sendOrder( const OrderFields & of )
    {
        // move end past SendingTime
        order.rewind( sendingTimeLength );
        // update changing fields in SendingTime
        order.sendingTime.update();
        // append order specific fields
        order.append<Account>( of.account, of.accountLen );
        order.append<ClOrdID>( of.orderId, of.orderIdLen );
        order.append<Symbol>( of.symbol, of.symbolLen );
        order.append<Side>( of.side );
        order.append<Price>( of.price, 6 );
        order.append<OrderQty>( of.qty );
        // copy SendingTime into TransactTime
        order.append<TransactTime>( order.sendingTime.begin, tsLen );
        order.append<OrdType>( of.type );
        // finalize
        order.setSeqnumAndUpdateHeaderAndChecksum(++seqnum);
        // send it
        socket.send( order.start, order.end - order.start );
    }

Examples

To compile the examples you wil have to clone the makefile project next to fixpp:

$> git clone https://github.com/sashamakarenko/makefile.git makefile
$> git clone https://github.com/sashamakarenko/fixpp.git fixpp
$> cd fixpp/examples/tiny
$> make
$> make check

• fix44 all committed complete lib with all FIX4.4 messages
• fixdump tool to decode FIX messages
• odd lib with unit tests with irregular messages
• tiny venue specific example with unit tests
• order all committed primitive project only with NewOrderSingle and ExecutionReport
• spec input files for different FIX standards

Custom HTML output

const FixFormatStyle htmlRgbStyle =
{
    .messageBegin =       "<pre>",
    .messageEnd =         "</pre>",
    .indent =             "  ",
    .groupFirstField =    " &#x2022;",
    .fieldBegin =         " ",
    .fieldEnd =           "\n",
    .headerTagNameStart = "<font color=\"#444444\">",
    .headerTagNameStop =  "</font>",
    .tagNameStart =       "<font color=\"black\"><b>",
    .tagNameStop =        "</b></font>",
    .tagValueStart =      "<font color=\"grey\">(",
    .tagValueStop =       ")</font>",
    .equal =              " = ",
    .valueStart =         "<font color=\"darkblue\">",
    .valueStop =          "</font>",
    .enumStart =          " <font color=\"darkgreen\">",
    .enumStop =           "</font>",
    .unknownStart =       "<font color=\"red\">",
    .unknownStop =        "</font>"
};
...

    std::ofstream html;
    html.open( "mdfr.html" );
    pos = 0;
    fixToHuman( mdFullRefresh, pos, html, htmlRgbStyle, autoIndentFields );
    html.close();

 BeginString(8) = FIX.4.4
 BodyLength(9) = 315
 MsgType(35) = W MARKET_DATA_SNAPSHOT_FULL_REFRESH
 SenderCompID(49) = foo
 TargetCompID(56) = bar
 MsgSeqNum(34) = 1234
 SendingTime(52) = 20190101-01:01:01.000
 Symbol(55) = EUR/USD
 NoMDEntries(268) = 6
   MDEntryType(269) = 1 OFFER
   MDEntryPositionNo(290) = 1
   MDEntryPx(270) = 1.21
   Currency(15) = USD
   MDEntrySize(271) = 1000000
   MDEntryType(269) = 1 OFFER
   MDEntryPositionNo(290) = 2
   MDEntryPx(270) = 1.211
   Currency(15) = USD
   MDEntrySize(271) = 2000000
   MDEntryType(269) = 1 OFFER
   MDEntryPositionNo(290) = 3
   MDEntryPx(270) = 1.221
   Currency(15) = USD
   MDEntrySize(271) = 3000000
   MDEntryType(269) = 1 OFFER
   MDEntryPositionNo(290) = 4
   MDEntryPx(270) = 1.2315
   Currency(15) = USD
   MDEntrySize(271) = 4000000
   MDEntryType(269) = 0 BID
   MDEntryPositionNo(290) = 5
   MDEntryPx(270) = 1.201
   Currency(15) = USD
   MDEntrySize(271) = 1000000
   MDEntryType(269) = 0 BID
   MDEntryPositionNo(290) = 6
   MDEntryPx(270) = 1.205
   Currency(15) = USD
   MDEntrySize(271) = 2000000
 CheckSum(10) = 075

Show message as HTML table:

const FixFormatStyle htmlTableRgbStyle =
{
    .messageBegin       = "<pre><table>",
    .messageEnd         = "</table></pre>",
    .indent             = "&nbsp;&nbsp;",
    .groupFirstField    = "&nbsp;&#x2022;",
    .fieldBegin         = "<tr><td>",
    .fieldEnd           = "</td></tr>\n",
    .headerTagNameStart = "<font color=\"#444444\">",
    .headerTagNameStop  = "</font>",
    .tagNameStart       = "<font color=\"black\"><b>",
    .tagNameStop        = "</b></font>",
    .tagValueStart      = "<font color=\"grey\">(",
    .tagValueStop       = ")</font>",
    .equal              = " </td><td> ",
    .valueStart         = "<font color=\"darkblue\">",
    .valueStop          = "</font>",
    .enumStart          = " <font color=\"darkgreen\">" ,
    .enumStop           = "</font>",
    .unknownStart       = "<font color=\"red\">",
    .unknownStop        = "</font>"
};

BeginString(8) 
 FIX.4.4
BodyLength(9) 
 332
MsgType(35) 
 8 EXECUTION_REPORT
SenderCompID(49) 
 foo
TargetCompID(56) 
 bar
SendingTime(52) 
 20071123-05:30:00.000
ClOrdID(11) 
 OID123456
ExecType(150) 
 E
OrdStatus(39) 
 A PENDING_NEW
Symbol(55) 
 XYZ
SecurityType(167) 
 CS COMMON_STOCK
Side(54) 
 1 BUY
OrderQty(38) 
 15
OrdType(40) 
 2 LIMIT
Price(44) 
 15.001
Text(58) 
 EQUITYTESTING
TimeInForce(59) 
 0 DAY
LastQty(32) 
 0
LastPx(31) 
 0
LeavesQty(151) 
 15
CumQty(14) 
 0
AvgPx(6) 
 0
NoLegs(555) 
 2
 •LegSymbol(600) 
 SYM1
  LegSide(624) 
 0
  LegQty(687) 
 10
  NoLegStipulations(683) 
 1
   •LegStipulationType(688) 
 A
    LegStipulationValue(689) 
 a
  LegPositionEffect(564) 
 1
  NoNestedPartyIDs(539) 
 2
   •NestedPartyID(524) 
 PARTY1
    NestedPartyIDSource(525) 
 S
   •NestedPartyID(524) 
 PARTY2
    NestedPartyIDSource(525) 
 S
    NoNestedPartySubIDs(804) 
 2
     •NestedPartySubID(545) 
 S1
      NestedPartySubIDType(805) 
 1
     •NestedPartySubID(545) 
 S2
      NestedPartySubIDType(805) 
 2
 •LegSymbol(600) 
 SYM2
  LegSide(624) 
 1
  LegQty(687) 
 20
  NoLegStipulations(683) 
 2
   •LegStipulationType(688) 
 A
    LegStipulationValue(689) 
 a
   •LegStipulationType(688) 
 B
    LegStipulationValue(689) 
 b
CheckSum(10) 
 027