atom_gotchas

An OTP application showcasing some atom gotchas related to Json decoding.

Build

    $ rebar3 compile

Gotchas

Json decoding atom bomb

Decoding Json keys as atoms by default can easily cause the number of atoms created by the runtime to be reached, which in turn causes Erlang VM to terminate.

1> erlang:system_info(atom_count).      
13020
2> atom_gotchas_app:generate_atoms(1000).
1000000 atoms generated
ok
3> erlang:system_info(atom_count).       
1013052
4> atom_gotchas_app:generate_atoms(100).
no more index entries in atom_tab (max=1048576)

Crash dump is being written to: erl_crash.dump...done

The limit of atoms of this particular runtime is 1048576. When the limit is reached, the Erlang VM terminates.

Json decoding with attempt_atom

Decoding Json keys using attempt_atom can have different results, when atom literals are used inside modules that don't get loaded at the application's startup.

Example:

1> A = atom_gotchas_app.
atom_gotchas_app
2> A:check(A:decode(A:json(8))).                           
atom1 NOT FOUND!!!
atom2 NOT FOUND!!!
atom3 NOT FOUND!!!
atom4 NOT FOUND!!!
atom5 found
atom6 found
atom7 found
atom8 NOT FOUND!!!
ok
3> A:check(A:decode(A:json(7))).
atom1 found
atom2 found
atom3 found
atom4 found
atom5 found
atom6 found
atom7 found
atom8 found
ok

The results above can be explained by the following facts:

1. atom1, atom2, atom3 and atom4 are literals used in module internal.erl, which only gets loaded when check/1 is called
2. atom5, atom6 and atom7 are literals used in module atom_gotchas_app.erl
3. atom8 is a dynamic atom, created inside internal:atom_exists/2 function

Atom reaper (tentative solution)

atom_reaper.escript parses the source files and extracts all atom literals.

$ escript atom_reaper.escript 
[info] Configured src paths: ["src"]
[info] Configured project app paths: ["apps/*","lib/*","."]
[info] App dirs: ["."]
[info] Processing app dir: "."
[info] Parsing file ./src/atom_gotchas_sup.erl
[info] Parsing file ./src/atom_gotchas_app.erl
[info] Parsing file ./src/internal.erl
[WARNING] Found a list_to_atom call in ./src/internal.erl:15 (function atom_exists)
[info] All atoms: [atom,atom1,atom1_exists,atom2,atom2_exists,atom3,
                   atom3_exists,atom4,atom4_exists,atom6,atom7,atom_exists,
                   atom_gotchas_sup,attempt_atom,decode,encode,false,foldl,
                   foreach,format,get,integer_to_list,intensity,internal,io,
                   is_key,json,jsone,keys,length,list_to_atom,list_to_binary,
                   lists,load_atom7,local,maps,ok,one_for_all,os,period,seq,
                   start_link,strategy,supervisor,system_time,true,undefined]

The literals are imported inside atom_gotchas_app.erl so that they get created when the application starts. As such, at the start of the application, all the atoms are already created:

1> A = atom_gotchas_app.
atom_gotchas_app
2>  A:check(A:decode(A:json(8))).
atom1 found
atom2 found
atom3 found
atom4 found
atom5 found
atom6 found
atom7 found
atom8 NOT FOUND!!!
ok

Functions description

• atom_gotchas_app:generate_atoms/1 generates a number of Jsons (equal to the integer received in input), each having 1K unique keys, decoding them with the option to always convert keys to atoms.

Example:

1> erlang:system_info(atom_count).     
263028
2> atom_gotchas_app:generate_atoms(7). 
7000 atoms generated
ok
3> erlang:system_info(atom_count).    
270028

• atom_gotchas_app:json/1 generates a string representing an encoded JSON of the form

{ 
    "atom1": 1,
    "atom2": 2,
    "atom3": 3,
    ...
}

Example:

1> atom_gotchas_app:json(3).
<<"{\"atom1\":1,\"atom2\":2,\"atom3\":3}">>

• atom_gotchas_app:decode/1 decodes an encoded Json, decoding keys to atom values (if they exist), or binary values, otherwise.

Example:

1> atom_gotchas_app:decode(<<"{\"atom1\":1,\"atom2\":2,\"atom3\":3}">>).
#{<<"atom1">> => 1,<<"atom2">> => 2,<<"atom3">> => 3}
2> atom1.
atom1
3> atom_gotchas_app:decode(<<"{\"atom1\":1,\"atom2\":2,\"atom3\":3}">>).
#{atom1 => 1,<<"atom2">> => 2,<<"atom3">> => 3}

• atom_gotchas_app:check/1 checks if the keys of a map generated by the previous 2 functions are atoms

1> atom_gotchas_app:check(#{atom1 => 1,<<"atom2">> => 2,<<"atom3">> => 3}).
atom1 found
atom2 NOT FOUND!!!
atom3 NOT FOUND!!!
ok