forked from mullerj/norm
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathNormDeveloperGuide.xml
7027 lines (5910 loc) · 338 KB
/
NormDeveloperGuide.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE article PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN"
"http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd">
<article>
<articleinfo>
<title><inlinemediaobject>
<imageobject>
<imagedata align="center" fileref="NormLogo.gif" scale="50"/>
</imageobject>
</inlinemediaobject> NORM Developer's Guide (version 1.5b5)</title>
<titleabbrev>NORM Developer's Guide</titleabbrev>
<abstract>
<para>This document describes an application programming interface (API)
for the <ulink url="https://www.nrl.navy.mil/Our-Work/Areas-of-Research/Information-Technology/NCS/NORM/">Nack-Oriented
Reliable Multicast (NORM)</ulink> protocol implementation developed by
the Protocol Engineering and Advance Networking (<ulink
url="https://www.nrl.navy.mil/itd/ncs/">PROTEAN</ulink>) Research Group of the
United States <ulink url="https://www.nrl.navy.mil/">Naval Research
Laboratory</ulink> (NRL). The NORM protocol provides general purpose
reliable data transport for applications wishing to use Internet
Protocol (IP) Multicast services for group data delivery. NORM can also
support unicast (point-to-point) data communication and may be used for
such when deemed appropriate. The current NORM protocol specification is
given in the <ulink url="https://www.ietf.org/">Internet Engineering Task
Force</ulink> (IETF) <ulink
url="https://datatracker.ietf.org/doc/html/rfc5740">RFC 5740</ulink>. This
document is currently a reference guide to the NORM API of the NRL
reference implementation. More tutorial material may be include in a
future version of this document or a separate developer's tutorial may
be created at a later date.</para>
</abstract>
</articleinfo>
<sect1>
<title>Background</title>
<para>This document describes an application programming interface (API)
for the <ulink url="https://www.nrl.navy.mil/Our-Work/Areas-of-Research/Information-Technology/NCS/NORM/">Nack-Oriented
Reliable Multicast (NORM)</ulink> protocol implementation developed by the
Protocol Engineering and Advance Networking (<ulink
url="https://www.nrl.navy.mil/itd/ncs/">PROTEAN</ulink>) Research Group of the
United States <ulink url="https://www.nrl.navy.mil/">Naval Research
Laboratory</ulink> (NRL). The NORM protocol provides general purpose
reliable data transport for applications wishing to use Internet Protocol
(IP) Multicast services for group data delivery. NORM can also support
unicast (point-to-point) data communication and may be used for such when
deemed appropriate. The current NORM protocol specification is given in
the <ulink url="https://www.ietf.org/">Internet Engineering Task
Force</ulink> (IETF) <ulink
url="https://datatracker.ietf.org/doc/html/rfc5740">RFC 5740</ulink>.</para>
<para>The NORM protocol is designed to provide end-to-end reliable
transport of bulk data objects or streams over generic IP multicast
routing and forwarding services. NORM uses a selective, negative
acknowledgement (NACK) mechanism for transport reliability and offers
additional protocol mechanisms to conduct reliable multicast sessions with
limited "a priori" coordination among senders and receivers. A congestion
control scheme is specified to allow the NORM protocol to fairly share
available network bandwidth with other transport protocols such as
Transmission Control Protocol (TCP). It is capable of operating with both
reciprocal multicast routing among senders and receivers and with
asymmetric connectivity (possibly a unicast return path) from the senders
to receivers. The protocol offers a number of features to allow different
types of applications or possibly other higher-level transport protocols
to utilize its service in different ways. The protocol leverages the use
of FEC-based repair and other proven reliable multicast transport
techniques in its design.</para>
<para>The NRL NORM library attempts to provide a general useful capability
for development of reliable multicast applications for bulk file or other
data delivery as well as support of stream-based transport with possible
real-time delivery requirements. The API allows access to many NORM
protocol parameters and control functions to tailor performance for
specific applications. While default parameters, where provided, can be
useful to a potential wide range of requirements, the many different
possible group communication paradigms dictate different needs for
different applications. Even with NORM, the developer should have a
thorough understanding of the specific application's group communication
needs.</para>
</sect1>
<sect1>
<title>Overview</title>
<para>The NORM API has been designed to provide simple, straightforward
access to and control of NORM protocol state and functions. Functions are
provided to create and initialize instances of the NORM API and associated
transport sessions (<emphasis>NormSessions</emphasis>). Subsequently, NORM
data transmission (<emphasis>NormSender</emphasis>) operation can be
activated and the application can queue various types of data
(<emphasis>NormObjects</emphasis>) for reliable transport. Additionally or
alternatively, NORM reception (<emphasis>NormReceiver</emphasis>)
operation can also be enabled on a per-session basis and the protocol
implementation alerts the application of receive events.</para>
<para>By default, the NORM API will create an operating system thread in
which the NORM protocol engine runs. This allows user application code and
the underlying NORM code to execute somewhat independently of one another.
The NORM protocol thread notifies the application of various protocol
events through a thread-safe event dispatching mechanism and API calls are
provided to allow the application to control NORM operation. (Note: API
mechanisms for lower-level, non-threaded control and execution of the NORM
protocol engine code may also be provided in the future.)</para>
<para>The NORM API operation can be roughly summarized with the following
categories of functions:</para>
<orderedlist>
<listitem>
<para>API Initialization</para>
</listitem>
<listitem>
<para>Session Creation and Control</para>
</listitem>
<listitem>
<para>Data Transport</para>
</listitem>
<listitem>
<para>API Event Notification</para>
</listitem>
</orderedlist>
<para>Note the order of these categories roughly reflects the order of
function calls required to use NORM in an application. The first step is
to create and initialize, as needed, at least one instance of the NORM
API. Then one or more NORM transport sessions (where a "session"
corresponds to data exchanges on a given multicast group (or unicast
address) and host port number) may be created and controlled. Applications
may participate as senders and/or receivers within a NORM session. NORM
senders transmit data to the session destination address (usually an IP
multicast group) while receivers are notified of incoming data. The NORM
API provides an event notification scheme to notify the application of
significant sender and receiver events. There are also a number support
functions provided for the application to control and monitor its
participation within a NORM transport session.</para>
<sect2>
<title>API Initialization</title>
<para>The NORM API requires that an application explicitly create at
least one instance of the NORM protocol engine that is subsequently used
as a conduit for further NORM API calls. By default, the NORM protocol
engine runs in its own operating system thread and interacts with the
application in a thread-safe manner through the API calls and event
dispatching mechanism.</para>
<para>In general, only a single thread should access the <link
linkend="NormGetNextEvent"><literal>NormGetNextEvent()</literal></link>
API call for a given <emphasis>NormInstance</emphasis>. This function
serves as the conduit for delivering NORM protocol engine events to the
application. A NORM application can be designed to be single-threaded,
even with multiple active NormSessions, but also multiple API instances
can be created (see <link
linkend="NormCreateInstance"><literal>NormCreateInstance()</literal></link>)
as needed for applications with specific requirements for accessing and
controlling participation in multiple <emphasis>NormSessions</emphasis>
from separate operating system threads. Or, alternatively, a
single <emphasis>NormInstance</emphasis> could be used, with a "master
thread" serving as an intermediary between the <link
linkend="NormGetNextEvent"><literal>NormGetNextEvent()</literal></link>
function, demultiplexing and dispatching events as appropriate to other
"child threads" that are created to handle
"per-<emphasis>NormSession</emphasis>" input/output. The advantage of
this alternative approach is that the end result would be one NORM
protocol engine thread plus one "master thread" plus one "child thread"
per <emphasis>NormSession</emphasis> instead of two threads (protocol
engine plus application thread) per <emphasis>NormSession</emphasis> if
such multi-threaded operation is needed by the application.</para>
</sect2>
<sect2>
<title>Session Creation and Control</title>
<para>Once an API instance has been successfully created, the
application may then create NORM transport session instances as needed.
The application can participate in each session as a sender and/or
receiver of data. If an application is participating as a sender, it may
enqueue data transport objects for transmission. The control of
transmission is largely left to the senders and API calls are provided
to control transmission rate, FEC parameters, etc. Applications
participating as receivers will be notified via the NORM API's event
dispatching mechanism of pending and completed reliable reception of
data along with other significant events. Additionally, API controls for
some optional NORM protocol mechanisms, such as positive acknowledgment
collection, are also provided.</para>
<para>Note when multiple senders are involved, receivers allocate system
resources (buffer space) for each active sender. With a very large
number of concurrently active senders, this may translate to significant
memory allocation on receiver nodes. Currently, the API allows the
application to control how much buffer space is allocated for each
active sender (NOTE: In the future, API functions may be provided to limit
the number of active senders monitored and/or provide the application
with finer control over receive buffer allocation, perhaps on a per
sender basis).</para>
</sect2>
<sect2>
<title>Data Transport</title>
<para>The NORM protocol supports transport of three basic types of data
content. These include the types <literal>NORM_OBJECT_FILE</literal> and
<literal>NORM_OBJECT_DATA</literal> which represent predetermined,
fixed-size application data content. The only differentiation with
respect to these two types is the implicit "hint" to the receiver to use
non-volatile (i.e. file system) storage or memory. This "hint" lets the
receiver allocate appropriate storage space with no other information on
the incoming data. The NORM implementation reads/writes data for the
<literal>NORM_OBJECT_FILE</literal> type directly from/to file storage,
while application memory space is accessed for the
<literal>NORM_OBJECT_DATA</literal> type. The third data content type,
<literal>NORM_OBJECT_STREAM</literal>, represents unbounded, possibly
persistent, streams of data content. Using this transport paradigm,
traditional, byte-oriented streaming transport service (e.g. similar to
that provided by a TCP socket) can be provided. Additionally, NORM has
provisions for application-defined message-oriented transport where
receivers can recover message boundaries without any "handshake" with
the sender. Stream content is buffered by the NORM implementation for
transmission/retransmission and as it is received.</para>
<sect3>
<title>Data Transmission</title>
<para>The behavior of data transport operation is largely placed in
the control of the NORM sender(s). NORM senders control their data
transmission rate, forward error correction (FEC) encoding settings,
and parameters controlling feedback from the receiver group. Multiple
senders may operate in a session, each with independent transmission
parameters. NORM receivers learn needed parameter values from fields
in NORM message headers.</para>
<para>NORM transport "objects" (file, data, or stream) are queued for
transmission by NORM senders. NORM senders may also cancel
transmission of objects at any time. The NORM sender controls the
transmission rate either manually (fixed transmission rate) or
automatically when NORM congestion control operation is enabled. The
NORM congestion control mechanism is designed to be "friendly" to
other data flows on the network, fairly sharing available
bandwidth.<link
linkend="NormSetAutoParity"><literal>NormSetAutoParity()</literal></link>)
to achieve reliable transfer) receive object transmission before any
extensive repair process that may be required to satisfy other
receivers with poor network connectivity. The repair boundary can also
be set for individual remote senders using the <link
linkend="NormNodeSetRepairBoundary"><literal>NormNodeSetRepairBoundary()</literal></link>
function.<literal>NORM_OBJECT_FILE</literal> objects. This function
must be called before any file objects may be received and thus should
be called before any calls to <link
linkend="NormStartReceiver"><literal>NormStartReceiver()</literal></link>
are made. However, note that the cache directory may be changed even
during active NORM reception. In this case, the newly specified
directory path will be used for subsequently-received files. Any files
received before a directory path change will remain in the previous
cache location. Note that the <link
linkend="NormFileRename"><literal>NormFileRename()</literal></link>
function may be used to rename, and thus potentially move, received
files after reception has begun.</para>
<para>By default, the NORM sender transmits application-enqueued data
content, providing repair transmissions (usually in the form of FEC
messages) only when requested by NACKs from the receivers. However,
the application may also configure NORM to proactively send some
amount of FEC content along with the original data content to create a
"robust" transmission that, in some cases, may be reliably received
without any NACKing activity. This can allow for some degree of
reliable protocol operation even without receiver feedback available.
NORM senders may also requeue (within the limits of "transmit cache"
settings) objects for repeat transmission, and receivers may combine
together multiple transmissions to reliably receive content.
Additionally, hybrid proactive/reactive FEC repair operation is
possible with the receiver NACK process as a "backup" for when network
packet loss exceeds the repair capability of the proactive FEC
settings.</para>
<para>The NRL NORM implementation also supports optional collection of
positive acknowledgment from a subset of the receiver group at
application-determined positions during data transmission. The NORM
API allows the application to specify the receiver subset ("acking
node list") and set "watermark" points for which positive
acknowledgement is collected. This process can provide the application
with explicit flow control for an application-determined critical set
of receivers in the group.</para>
<para>For a NORM application to perform data transmission, it must
first create a session using <link
linkend="NormCreateSession"><literal>NormCreateSession()</literal></link>
and make a call to <link
linkend="NormStartSender"><literal>NormStartSender()</literal></link>
before sending actual user data. The functions <link
linkend="NormFileEnqueue"><literal>NormFileEnqueue()</literal></link>,
<link
linkend="NormDataEnqueue"><literal>NormDataEnqueue()</literal></link>,
and <link
linkend="NormStreamWrite"><literal>NormStreamWrite()</literal></link>
are available for the application to pass data to the NORM protocol
engine for transmission. Note that to use <link
linkend="NormStreamWrite"><literal>NormStreamWrite()</literal></link>,
a "sender stream" must first be created using <link
linkend="NormStreamOpen"><literal>NormStreamOpen()</literal></link>.
In the case of <link
linkend="NormFileEnqueue"><literal>NormFileEnqueue()</literal></link>
and <link
linkend="NormDataEnqueue"><literal>NormDataEnqueue()</literal></link>,
the NORM protocol engine directly accesses the application file or
memory space to refer to the transmitted content and does not make its
own copy of this data.</para>
<para>The calls to enqueue transport objects or write to a stream may
be called at any time, but the <literal>NORM_TX_QUEUE_EMPTY</literal>
and <literal>NORM_TX_QUEUE_VACANCY</literal> notification events (see
<link
linkend="NormGetNextEvent"><literal>NormGetNextEvent()</literal></link>)
provide useful cues for when these functions may be successfully
called. Typically, an application might catch both
<literal>NORM_TX_QUEUE_EMPTY</literal> and
<literal>NORM_TX_QUEUE_VACANCY</literal> event types as cues for
enqueuing additional transport objects or writing to a stream.
However, an application may choose to cue off of
<literal>NORM_TX_QUEUE_EMPTY</literal> only if it wishes to provide
the "freshest" data to NORM for transmission. The advantage of
additionally using <literal>NORM_TX_QUEUE_VACANCY</literal> is that if
the application uses this cue to fill up NORM transport object or
stream buffers, it can keep the NORM stream busy sending data and
realize the highest possible transmission rate when attempting very
high speed communication (Otherwise, the NORM protocol engine may
experience some "dead air time" waiting for the application thread to
respond to a <literal>NORM_TX_QUEUE_EMPTY</literal> event). Note the
sender application can control buffer depths as needed with the <link
linkend="NormSetTxCacheBounds"><literal>NormSetTxCacheBounds()</literal></link>
and <link
linkend="NormStreamOpen"><literal>NormStreamOpen()</literal></link>
calls. Additionally, it is possible for applications to configure the
transmit object "cache" (see <link
linkend="NormSetTxCacheBounds"><literal>NormSetTxCacheBounds()</literal></link>)
and use the <link
linkend="NormRequeueObject"><literal>NormRequeueObject()</literal></link>
call (for objects that have not yet received a
<parameter>NORM_TX_OBJECT_PURGED</parameter> notification) to effect a
sort of "data carousel" operation with repeated transmission of the
cached objects. The <parameter>NORM_TX_OBJECT_SENT</parameter>
notification can be used a cue to properly control the "requeue"
cycle(s).</para>
<para>The NORM implementation provides a form of timer-based flow
control that limits how quickly sender applications may enqueue new
objects or stream data for transmission. The <link
linkend="NormSetFlowControl"><literal>NormSetFlowControl()</literal></link>
call is provided to control this behavior, including the option to
disable it. This timer-based mechanism is a type of "soft" flow
control by allowing receivers "sufficient" time to request repair of
pending data the sender has enqueued. A more explicit form of flow
control using the optional "watermark flushing" mechanism is described
below.</para>
<para>Another cue that can be leveraged by the sender application to
determine when it is appropriate to enqueue (or write) additional data
for transmission is the <literal>NORM_TX_WATERMARK_COMPLETED</literal>
event. This event is posted when the flushing or explicit positive
acknowledgment collection process has completed for a "watermark"
point in transmission that was set by the sender (see <link
linkend="NormSetWatermark"><literal>NormSetWatermark()</literal></link>
and <link
linkend="NormAddAckingNode"><literal>NormAddAckingNode()</literal></link>).
A list of <link
linkend="NormNodeId"><literal>NormNodeId</literal></link> values can
be supplied from which explicit acknowledgement is expected and/or the
<link linkend="NormNodeId"><literal>NormNodeId</literal></link>
<literal>NORM_NODE_NONE</literal> can be set (using <link
linkend="NormAddAckingNode"><literal>NormAddAckingNode()</literal></link>)
for completion of a NACK-based version of the watermark flushing
procedure. This flushing process can be used as a flow control
mechanism for NORM applications. Note this is distinct from NORM's
congestion control mechanism that, while it provides network-friendly
transmission rate control, does guarantee flow control to receiving
nodes.<literal>NORM_NODE_NONE</literal> can be set (using <link
linkend="NormAddAckingNode"><literal>NormAddAckingNode()</literal></link>)
for completion of a NACK-based version of the watermark flushing
procedure. This flushing process can be used as a flow control
mechanism for NORM applications. Note this is distinct from NORM's
congestion control mechanism that, while it provides network-friendly
transmission rate control, does guarantee flow control to receiving
nodes.</para>
</sect3>
<sect3>
<title>Data Reception</title>
<para>NORM receiver applications learn of active senders and their
corresponding pending and completed data transfers, etc via the API
event dispatching mechanism. By default, NORM receivers use NACK
messages to request repair of transmitted content from the originating
sender as needed to achieve reliable transfer. Some API functions are
available to provide some additional control over the NACKing
behavior, such as initially NACKing for <literal>NORM_INFO</literal>
content only or even to the extent of disabling receiver feedback
(silent receiver or emission-controlled (EMCON) operation) entirely.
Otherwise, the parameters and operation of reliable data transmission
are left to sender applications and receivers learn of sender
parameters in NORM protocol message headers and are instructed by
<literal>NORM_CMD</literal> messages from the sender(s).</para>
<para>With respect to the NORM API, the receiver application is
informed of new senders and receive data objects via the the
<parameter>NORM_REMOTE_SENDER_NEW</parameter> and
<parameter>NORM_RX_OBJECT_NEW</parameter> notifications, respectfully.
Additionally, object reception progress is indicated with the
<parameter>NORM_RX_OBJECT_UPDATED</parameter> notification and this
also serves as an indicator for the
<parameter>NORM_OBJECT_STREAM</parameter> type that the receive
application should make calls to <link
linkend="NormStreamRead"><literal>NormStreamRead()</literal></link> to
read newly received stream content. NORM sender status is also
conveyed via the <parameter>NORM_REMOTE_SENDER_ACTIVE</parameter> and
NORM_REMOTE_SENDER_INACTIVE notifications. For example, the receiver
application may use the
<parameter>NORM_REMOTE_SENDER_INACTIVE</parameter> as a cue to make
calls to <link
linkend="NormNodeFreeBuffers"><literal>NormNodeFreeBuffers()</literal></link>
and/or <link
linkend="NormNodeDelete"><literal>NormNodeDelete()</literal></link> to
free memory resources allocated for buffering received content for the
given sender. The amount of memory allocated <emphasis>per
sender</emphasis> is set in the <link
linkend="NormStartReceiver"><literal>NormStartReceiver()</literal></link>
call.</para>
</sect3>
</sect2>
<sect2>
<title>API Event Notification</title>
<para>An asynchronous event dispatching mechanism is provided to notify
the application of significant NORM protocol events. The centerpiece of
this is the <link
linkend="NormGetNextEvent"><literal>NormGetNextEvent()</literal></link>
function that can be used to retrieve the next NORM protocol engine
event in the form of a <link
linkend="NormEvent"><literal>NormEvent</literal></link> structure. This
function will typically block until a <link
linkend="NormEvent"><literal>NormEvent</literal></link> occurs. However,
non-blocking operation may be achieved by using the <link
linkend="NormGetDescriptor"><literal>NormGetDescriptor()</literal></link>
call to get a <link
linkend="NormDescriptor"><type>NormDescriptor</type></link> (file
descriptor) value (Unix <type>int</type> or Win32 <type>HANDLE</type>)
suitable for use in a asynchronous I/O monitoring functions such as the
Unix <function>select()</function> or Win32
<function>MsgWaitForMultipleObjects()</function> system calls. The a
<link linkend="NormDescriptor"><type>NormDescriptor</type></link> will
be signaled when a <link
linkend="NormEvent"><literal>NormEvent</literal></link> is available.
For Win32 platforms, dispatching of a user-defined Windows message for
NORM event notification is also planned for a future update to the NORM
API.</para>
</sect2>
</sect1>
<sect1>
<title>Build Notes</title>
<para>To build applications that use the NORM library, a path to the
"normApi.h" header file must be provided and the linker step needs to
reference the NORM library file ("<filename>libnorm.a</filename>" for Unix
platforms and "<filename>Norm.lib</filename>" for Win32 platforms). NORM
also depends upon the NRL Protean Protocol Prototyping toolkit "Protokit"
library (a.k.a "Protolib") (static library files
"<filename>libProtokit.a</filename>" for Unix and
"<filename>Protokit.lib</filename>" for Win32). Shared or
dynamically-linked versions of these libraries may also be built from the
NORM source code or provided. Depending upon the platform, some additional
library dependencies may be required to support the needs of NORM and/or
Protokit. These are described below.</para>
<para>The "makefiles" directory contains Unix Makefiles for various
platforms. The "win32" and "wince" sub-directories there contain Microsoft
Visual C++ (VC++) and Embedded VC++ project files for building the NORM
implementation. Additionally, a "waf" (Python-based build tool) build
option is supported that can be used to build and install the NORM library
code on the supported platforms. Finally, Python and Java bindings to the
NORM API are included in the "src/python" and "src/java" directories and the
"makefiles/java" directory contains Makefiles to build the NORM Java JNI bindings.
Note the "waf" tool can also be used to build the Java and Python bindings.</para>
<sect2>
<title>Unix Platforms</title>
<para>NORM has been built and tested on Linux (various architectures),
MacOS (BSD), Solaris, and IRIX (SGI) platforms. The code should be
readily portable to other Unix platforms.</para>
<para>To support IPv6 operation, the NORM and the Protokit library must
be compiled with the "<constant>HAVE_IPV6</constant>" macro defined.
This is default in the NORM and Protokit Makefiles for platforms that
support IPv6. It is important that NORM and Protokit be built with this
macro defined the same. With NORM, it is recommended that "large file
support" options be enabled when possible.</para>
<para>The NORM API uses threading so that the NORM protocol engine may
run independent of the application. Thus the "POSIX Threads" library
must be included ("-pthread") in the linking step. MacOS/BSD also
requires the addition of the "-lresolv" (resolver) library and Solaris
requires the dynamic loader, network/socket, and resolver libraries
("-lnsl -lsocket -lresolv") to achieve successful compilation. The
Makefiles in the NORM source code distribution are a reference for these
requirements. Note that MacOS 9 and earlier are not supported.</para>
<para>Additionally, it is critical that the
_<constant>FILE_OFFSET_BITS</constant> macro be consistently defined for
the NORM library build and the application build using the library. The
distributed NORM Makefiles have
<constant>-D_FILE_OFFSET_BITS=64</constant> set in the compilation to
enable "large file support". Applications built using NORM should have
the same compilation option set to operate correctly (The definition of
the <link linkend="NormSize"><literal>NormSize</literal></link> type in
"<filename>normApi.h</filename>" depends upon this compilation
flag).</para>
</sect2>
<sect2>
<title>Win32/WiNCE Platforms</title>
<para>NORM has been built using Microsoft's Visual C++ (6.0 and .NET)
and Embedded VC++ 4.2 environments. In addition to proper macro
definitions (e.g., HAVE_IPV6, etc) that are included in the respective
"Protokit" and "NORM" project files, it is important that common code
generation settings be used when building the NORM application. The NORM
and Protokit projects are built with the "Multi-threading DLL" library
usage set. The NORM API requires multi-threading support. This is a
critical setting and numerous compiler and linker errors will result if
this is not properly set for your application project.</para>
<para>NORM and Protokit also depend on the Winsock 2.0
("<filename>ws2_32.lib</filename>" (or "<filename>ws2.lib</filename>"
(WinCE)) and the IP Helper API ("<filename>iphlpapi.lib</filename>")
libraries and these must be included in the project "Link"
attributes.</para>
<para>An additional note is that a bug in VC++ 6.0 and earlier compilers
(includes embedded VC++ 4.x compilers) prevent compilation of
Protokit-based code with debugging capabilities enabled. However, this
has been resolved in VC++ .NET and is hoped to be resolved in the future
for the WinCE build tools.</para>
<para>Operation on Windows NT4 (and perhaps other older Windows
operating systems) requires that the compile time macro
<constant>WINVER=0x0400</constant> defined. This is because the version
of the IP Helper API library (<filename>iphlpapi.lib</filename>) used by
<emphasis>Protolib</emphasis> (and hence NORM) for this system doesn't
support some of the functions defined for this library. This may be
related to IPv6 support issues so it may be possible that the Protolib
build could be tweaked to provide a single binary executable suitable
for IPv4 operation only across a large range of Windows
platforms.</para>
</sect2>
</sect1>
<sect1>
<title>API Reference</title>
<para>This section provides a reference to the NORM API variable types,
constants and functions.</para>
<sect2>
<title>API Variable Types and Constants</title>
<para>The NORM API defines and enumerates a number of supporting
variable types and values which are used in different function calls.
The variable types are described here.</para>
<sect3 id="NormInstanceHandle">
<title>NormInstanceHandle</title>
<para>The <link
linkend="NormInstanceHandle"><literal>NormInstanceHandle</literal></link>
type is returned when a NORM API instance is created (see <link
linkend="NormCreateInstance"><literal>NormCreateInstance()</literal></link>).
This handle can be subsequently used for API calls which require
reference to a specific NORM API instance. By default, each NORM API
instance instantiated creates an operating system thread for protocol
operation. Note that multiple NORM transport sessions may be created
for a single API instance. In general, it is expected that
applications will create a single NORM API instance, but some
multi-threaded application designs may prefer multiple corresponding
NORM API instances. The value <literal>NORM_INSTANCE_INVALID</literal>
corresponds to an invalid API instance.</para>
</sect3>
<sect3 id="NormSessionHandle">
<title>NormSessionHandle</title>
<para>The <link
linkend="NormSessionHandle"><literal>NormSessionHandle</literal></link>
type is used to reference NORM transport sessions which have been
created using the <link
linkend="NormCreateSession"><literal>NormCreateSession()</literal></link>
API call. Multiple <link
linkend="NormSessionHandle"><literal>NormSessionHandle</literal></link>
values may be associated with a given <link
linkend="NormInstanceHandle"><literal>NormInstanceHandle</literal></link>.
The special value <literal>NORM_SESSION_INVALID</literal> is used to
refer to invalid session references.</para>
</sect3>
<sect3 id="NormSessionId">
<title>NormSessionId</title>
<para>The <link
linkend="NormSessionId"><literal>NormSessionId</literal></link> type
is used by applications to uniquely identify their instance of
participation as a sender within a <emphasis>NormSession</emphasis>.
This type is a parameter to the <link
linkend="NormStartSender"><literal>NormStartSender()</literal></link>
function. Robust applications can use different <link
linkend="NormSessionId"><literal>NormSessionId</literal></link> values
when initiating sender operation so that receivers can discriminate
when a sender has terminated and restarted (whether intentional or due
to system failure). For example, an application could cache its prior
<link linkend="NormSessionId"><literal>NormSessionId</literal></link>
value in non-volatile storage which could then be recovered and
incremented (for example) upon system restart to produce a new value.
The <link
linkend="NormSessionId"><literal>NormSessionId</literal></link> value
is used for the value of the instance_id field in NORM protocol sender
messages (see the NORM protocol specification) and receivers use this
field to detect sender restart within a
<emphasis>NormSession</emphasis>.</para>
</sect3>
<sect3 id="NormNodeHandle">
<title>NormNodeHandle</title>
<para>The <link
linkend="NormNodeHandle"><literal>NormNodeHandle</literal></link> type
is used to reference state kept by the NORM implementation with
respect to other participants within a
<emphasis>NormSession</emphasis>. Most typically, the <link
linkend="NormNodeHandle"><literal>NormNodeHandle</literal></link> is
used by receiver applications to dereference information about remote
senders of data as needed. The special value
<literal>NORM_NODE_INVALID</literal> corresponds to an invalid
reference.</para>
</sect3>
<sect3 id="NormNodeId">
<title>NormNodeId</title>
<para>The <link
linkend="NormNodeId"><literal>NormNodeId</literal></link> type
corresponds to a 32-bit numeric value which should uniquely identify a
participant (node) in a given <emphasis>NormSession</emphasis>. The
<link
linkend="NormNodeGetId"><literal>NormNodeGetId()</literal></link>
function can be used to retrieve this value given a valid <link
linkend="NormNodeHandle"><literal>NormNodeHandle</literal></link>. The
special value <literal>NORM_NODE_NONE</literal> corresponds to an
invalid (or null) node while the value
<literal>NORM_NODE_ANY</literal> serves as a wild card value for some
functions.</para>
</sect3>
<sect3 id="NormObjectHandle">
<title>NormObjectHandle</title>
<para>The <link
linkend="NormObjectHandle"><literal>NormObjectHandle</literal></link>
type is used to reference state kept for data transport objects being
actively transmitted or received. The state kept for NORM transport
objects is temporary, but the NORM API provides a function to
persistently retain state associated with a sender or receiver <link
linkend="NormObjectHandle"><literal>NormObjectHandle</literal></link>
(see <link
linkend="NormObjectRetain"><literal>NormObjectRetain()</literal></link>)
if needed. For sender objects, unless explicitly retained, the <link
linkend="NormObjectHandle"><literal>NormObjectHandle</literal></link>
can be considered valid until the referenced object is explicitly
canceled (see <link
linkend="NormObjectCancel"><literal>NormObjectCancel()</literal></link>)
or purged from the sender transmission queue (see the event
<literal>NORM_TX_OBJECT_PURGED</literal>). For receiver objects, these
handles should be treated as valid only until a subsequent call to
<link
linkend="NormGetNextEvent"><literal>NormGetNextEvent()</literal></link>
unless, again, specifically retained. The special value
<literal>NORM_OBJECT_INVALID</literal> corresponds to an invalid
transport object reference.</para>
</sect3>
<sect3 id="NormObjectType">
<title>NormObjectType</title>
<para>The <link
linkend="NormObjectType"><literal>NormObjectType</literal></link> type
is an enumeration of possible NORM data transport object types. As
previously mentioned, valid types include:</para>
<orderedlist>
<listitem>
<para><literal>NORM_OBJECT_FILE</literal></para>
</listitem>
<listitem>
<para><literal>NORM_OBJECT_DATA</literal>, and</para>
</listitem>
<listitem>
<para><literal>NORM_OBJECT_STREAM</literal></para>
</listitem>
</orderedlist>
<para>Given a <link
linkend="NormObjectHandle"><literal>NormObjectHandle</literal></link>,
the application may determine an object's type using the <link
linkend="NormObjectGetType"><literal>NormObjectGetType()</literal></link>
function call. A special <link
linkend="NormObjectType"><literal>NormObjectType</literal></link>
value, <literal>NORM_OBJECT_NONE</literal>, indicates an invalid
object type.</para>
</sect3>
<sect3 id="NormSize">
<title>NormSize</title>
<para>The <link linkend="NormSize"><literal>NormSize</literal></link>
is the type used for <emphasis>NormObject</emphasis> size information.
For example, the <link
linkend="NormObjectGetSize"><literal>NormObjectGetSize()</literal></link>
function returns a value of type <link
linkend="NormSize"><literal>NormSize</literal></link>. The range of
<link linkend="NormSize"><literal>NormSize</literal></link> values
depends upon the operating system and NORM library compilation
settings. With "large file support" enabled, as is the case with
distributed NORM library "Makefiles", the <link
linkend="NormSize"><literal>NormSize</literal></link> type is a 64-bit
integer. However, some platforms may support only 32-bit object
sizes.</para>
</sect3>
<sect3 id="NormObjectTransportId">
<title>NormObjectTransportId</title>
<para>The <link
linkend="NormObjectTransportId"><literal>NormObjectTransportId</literal></link>
type is a 16-bit numerical value assigned to
<emphasis>NormObjects</emphasis> by senders during active transport.
These values are temporarily unique with respect to a given sender
within a <emphasis>NormSession</emphasis> and may be "recycled" for
use for future transport objects. NORM sender nodes assign these
values in a monotonically increasing fashion during the course of a
session as part of protocol operation. Typically, the application
should not need access to these values, but an API call such as
<function>NormObjectGetTransportId()</function>
(<emphasis>TBD</emphasis>) may be provided to retrieve these values if
needed. (Note this type may be deprecated; i.e., it may not be needed
at since the <link
linkend="NormRequeueObject"><function>NormRequeueObject()</function></link>
function is implemented using handles only, but _some_ applications
requiring persistence even after a system reboot may need the ability
to recall previous transport ids?)</para>
</sect3>
<sect3 id="NormEventType">
<title>NormEventType</title>
<para>The <link
linkend="NormEventType"><literal>NormEventType</literal></link> is an
enumeration of NORM API events. "Events" are used by the NORM API to
signal the application of significant NORM protocol operation events
(e.g., receipt of a new receive object, etc). A description of
possible <link
linkend="NormEventType"><literal>NormEventType</literal></link> values
and their interpretation is given below. The function call <link
linkend="NormGetNextEvent"><literal>NormGetNextEvent()</literal></link>
is used to retrieve events from the NORM protocol engine.</para>
</sect3>
<sect3 id="NormEvent">
<title>NormEvent</title>
<para>The <link
linkend="NormEvent"><literal>NormEvent</literal></link> type is a
structure used to describe significant NORM protocol events. This
structure is defined as follows:</para>
<para><programlisting>typedef struct
{
NormEventType type;
<link linkend="NormSessionHandle"><literal>NormSessionHandle</literal></link> session;
<link linkend="NormNodeHandle"><literal>NormNodeHandle</literal></link> sender;
<link linkend="NormObjectHandle"><literal>NormObjectHandle</literal></link> object;
} <link linkend="NormEvent"><literal>NormEvent</literal></link>;</programlisting></para>
<para>The <parameter>type</parameter> field indicates the <link
linkend="NormEventType"><literal>NormEventType</literal></link> and
determines how the other fields should be interpreted. Note that not
all <link
linkend="NormEventType"><literal>NormEventType</literal></link> fields
are relevant to all events. The <parameter>session</parameter>,
<parameter>sender</parameter>, and <parameter>object</parameter> fields
indicate the applicable <link
linkend="NormSessionHandle"><literal>NormSessionHandle</literal></link>,
<link
linkend="NormNodeHandle"><literal>NormNodeHandle</literal></link>, and
<link
linkend="NormObjectHandle"><literal>NormObjectHandle</literal></link>,
respectively, to which the event applies. NORM protocol events are
made available to the application via the <link
linkend="NormGetNextEvent"><literal>NormGetNextEvent()</literal></link>
function call.</para>
</sect3>
<sect3 id="NormDescriptor">
<title>NormDescriptor</title>
<para>The <link
linkend="NormDescriptor"><literal>NormDescriptor</literal></link> type
can provide a reference to a corresponding file descriptor (Unix
<type>int</type> or Win32 <type>HANDLE</type>) for the
<emphasis>NormInstance</emphasis>. For a given <link
linkend="NormInstanceHandle"><literal>NormInstanceHandle</literal></link>,
the <link
linkend="NormGetDescriptor"><literal>NormGetDescriptor()</literal></link>
function can be used to retrieve a <link
linkend="NormDescriptor"><literal>NormDescriptor</literal></link>
value that may, in turn, used in appropriate system calls (e.g.
<function>select()</function> or
<function>MsgWaitForMultipleObjects()</function>) to asynchronously
monitor the NORM protocol engine for notification events (see <link
linkend="NormEvent"><literal>NormEvent</literal></link>
description).</para>
</sect3>
<sect3 id="NormFlushMode">
<title>NormFlushMode</title>
<para>The <link
linkend="NormFlushMode"><literal>NormFlushMode</literal></link> type
consists of the following enumeration:</para>
<para><programlisting>enum <link linkend="NormFlushMode"><literal>NormFlushMode</literal></link>
<literal>{
NORM_FLUSH_NON</literal>E<literal>,
NORM_FLUSH_PASSIV</literal>E<literal>,
NORM_FLUSH_ACTIV</literal>E
};</programlisting></para>
<para>The use and interpretation of these values is given in the
descriptions of <link
linkend="NormStreamFlush"><literal>NormStreamFlush()</literal></link>
and <link
linkend="NormStreamSetAutoFlush"><literal>NormStreamSetAutoFlush()</literal></link>
functions.</para>
</sect3>
<sect3 id="NormProbingMode">
<title>NormProbingMode</title>
<para>The <link
linkend="NormProbingMode"><literal>NormProbingMode</literal></link>
type consists of the following enumeration:</para>
<para><programlisting>enum <link linkend="NormProbingMode"><literal>NormProbingMode</literal></link>
{
<literal>NORM_PROBE_NON</literal>E<literal>,
NORM_PROBE_PASSIV</literal>E<literal>,
NORM_PROBE_ACTIV</literal>E
};</programlisting></para>
<para>The use and interpretation of these values is given in the
description of <link
linkend="NormSetGrttProbingMode"><literal>NormSetGrttProbingMode()</literal></link>
function.</para>
</sect3>
<sect3 id="NormSyncPolicy">
<title>NormSyncPolicy</title>
<para>The <link
linkend="NormSyncPolicy"><literal>NormSyncPolicy</literal></link> type
consists of the following enumeration:</para>
<para><programlisting>enum <link linkend="NormSyncPolicy"><literal>NormSyncPolicy</literal></link>
{
<literal>NORM_SYNC_CURRENT</literal><literal>,
NORM_</literal>SYNC_ALL<literal>
};</literal></programlisting></para>
<para>The use and interpretation of these values is given in the
descriptions of the <link
linkend="NormSetDefaultSyncPolicy"><literal>NormSetDefaultSyncPolicy()</literal></link>
function.</para>
</sect3>
<sect3 id="NormNackingMode">
<title>NormNackingMode</title>
<para>The <link
linkend="NormNackingMode"><literal>NormNackingMode</literal></link>
type consists of the following enumeration:</para>
<para><programlisting>enum <link linkend="NormNackingMode"><literal>NormNackingMode</literal></link>
{
<literal>NORM_NACK_NON</literal>E<literal>,
NORM_NACK_INFO_ONL</literal>Y,
<literal>NORM_NACK_NORMAL
};</literal></programlisting></para>
<para>The use and interpretation of these values is given in the
descriptions of the <link
linkend="NormSetDefaultNackingMode"><literal>NormSetDefaultNackingMode()</literal></link>,
<link
linkend="NormNodeSetNackingMode"><literal>NormNodeSetNackingMode()</literal></link>
and <link
linkend="NormObjectSetNackingMode"><literal>NormObjectSetNackingMode()</literal></link>
functions.</para>
</sect3>
<sect3 id="NormRepairBoundary">
<title>NormRepairBoundary</title>
<para>The <link
linkend="NormRepairBoundary"><literal>NormRepairBoundary</literal></link>
types consists of the following enumeration:</para>
<para><programlisting>enum <link linkend="NormRepairBoundary"><literal>NormRepairBoundary</literal></link>
{<literal>
NORM_BOUNDARY_BLOC</literal>K,
N<literal>ORM_BOUNDARY_OBJECT
};</literal></programlisting></para>
<para>The interpretation of these values is given in the descriptions
of the <link
linkend="NormSetDefaultRepairBoundary"><literal>NormSetDefaultRepairBoundary()</literal></link>
and <link
linkend="NormNodeSetRepairBoundary"><literal>NormNodeSetRepairBoundary()</literal></link>
functions.</para>
</sect3>
<sect3 id="NormAckingStatus">
<title>NormAckingStatus</title>
<para>The <link
linkend="NormAckingStatus"><literal>NormAckingStatus</literal></link>
consist of the following enumeration:</para>
<para><programlisting>enum <link linkend="NormAckingStatus"><literal>NormAckingStatus</literal></link>
{
<literal>NORM_ACK_INVALID</literal>,<literal>
NORM_ACK_FAILUR</literal>E<literal>,
NORM_ACK_PENDIN</literal>G,
<literal>NORM_ACK_SUCCES</literal>S
};</programlisting></para>
<para>The interpretation of these values is given in the descriptions
of the <link
linkend="NormGetAckingStatus"><literal>NormGetAckingStatus()</literal></link>
function.</para>
</sect3>
</sect2>
<sect2>
<title>API Initialization and Operation</title>
<para>The first step in using the NORM API is to create an "instance" of
the NORM protocol engine. Note that multiple instances may be created by
the application if necessary, but generally only a single instance is
required since multiple <emphasis>NormSessions</emphasis> may be managed
under a single NORM API instance.</para>
<sect3 id="NormCreateInstance">
<title>NormCreateInstance()</title>
<sect4>
<title>Synopsis</title>
<programlisting>#include <normApi.h>
<link linkend="NormInstanceHandle"><literal>NormInstanceHandle</literal></link> <link
linkend="NormCreateInstance"><literal>NormCreateInstance</literal></link>(bool <parameter>priorityBoost</parameter> = false);</programlisting>
</sect4>
<sect4>
<title>Description</title>
<para>This function creates an instance of a NORM protocol engine
and is the necessary first step before any other API functions may
be used. With the instantiation of the NORM protocol engine, an
operating system thread is created for protocol execution. The
returned <link
linkend="NormInstanceHandle"><literal>NormInstanceHandle</literal></link>
value may be used in subsequent API calls as needed, such <link
linkend="NormCreateSession"><literal>NormCreateSession()</literal></link>,
etc. The optional <parameter>priorityBoost</parameter> parameter,
when set to a value of true, specifies that the NORM protocol engine
thread be run with higher priority scheduling. On Win32 platforms,
this corresponds to