Overclock.net › Forums › Software, Programming and Coding › Coding and Programming › Data Payload Decoding - Issues with Byte Reading
New Posts  All Forums:Forum Nav:

Data Payload Decoding - Issues with Byte Reading

post #1 of 30
Thread Starter 
Hello all,

I am wondering if anyone can provide a bit of guidance to help me understand a bit better on what path to go down.

I have written a packet sniffer program in C++ using Winsock in order to receive all IP packets on my Ethernet link. I will be filtering these packets out based on port 161 and 162 (traps).

Currently it can successfully extract fields from the IP header and also TCP/UDP header; also I am dumping these to a text file. The hex values and also ASCII values of the headers and data payload field are outputted.


However here is my dilemma, how do I go about decoding a specific protocol? In my case SNMP. Do I need to delve into basic encoding rules or abstract syntax notation 1 in order to understand these packets?


I am going off this image as a reference HERE. I know their hex values but I need to get them into human readable form.

Also I will be looking at decoding those OIDs.


Note: I am not looking at additional libraries I want to get this done in native C++ with Winsock if possible.


I haven't really a clue how to go about this; I've got Cisco behind me but this is my first time embarking on a programming based project. I've gained some good C++ knowledge so it's just a matter of "How do I go about decoding them?"

Any help is appreciated!! smile.gif


Update and Edit:

Having a new problem now. It is working with bytes.

My program will receive data off an interface (SIO_RECVALL) and throw it into an unsigned char array; i.e. byte array.

Now, when it comes to sifting through the data my program outputs garbled data for only bytes whose value is greater than 127.

I am outputting using unsigned int; so I am not sure.

Example:
if the bit pattern is 0111 1111 it's fine, 127.
If the bit pattern is 1000 0000 it outputs something on the lines of 4294967xxx, where X is varying.

I am confused, why shouldn't it just output the data, i.e the hexadecimal value 82 is 130 in decimal, and I get 4294967170 for it.

The bit pattern is 1000 0010; so why is my program having issues with it?

Regards,
Mike
Edited by mikeo01 - 4/20/14 at 6:04am
post #2 of 30
Have you just tried raw conversion to integer base 16 and then back to Unicode characters to see if it is actually encoded as? I know nothing really of encoding/decoding. So I assumed you tried converting to plain text format.

If doing this through Windows / VSStudio then you have .NET Framework, there is an opensource class with a .ToString() built in.
http://www.docs.snmpsharpnet.com/docs-0-5-0/html/N_SnmpSharpNet.htm

Don't re-invent the wheel if you don't have to.
Snowdevil
(16 items)
 
ASUS G750JM
(9 items)
 
 
CPUMotherboardGraphicsGraphics
[i7 4790K @ 4.4 GHz (1.186v)] [Asus Sabertooth Z97 Mark S] [nVidia Geforce GTX 1080] [nVidia Geforce GTX 1080] 
RAMHard DriveCoolingOS
[G.Skill 32GB DDR3 2133 MHz] [Crucial MX100 256GB] [Phanteks PH-TC12DX] [Win 10.1 Pro] 
MonitorMonitorKeyboardPower
[LG 29UM65 (2560x1080)] [QNIX Evo II LED (2560x1440)] [WASD v2 Tenkeyless] [NZXT Hale90 v2 ] 
CaseMouseMouse PadAudio
[ThermalTake GT10 Snow Edition] [Razer Mamba - Chroma] [Razer Kabuto] [Razer Man O' War] 
CPUMotherboardGraphicsRAM
i7 4770HQ Intel HM87 Express Chipset Geforce GTX 860M 8GB DDR3L 1600 MHz 
Hard DriveOptical DriveCoolingOS
Samsung SSD EVO DVD-RW Stock Windows 8.1 
Monitor
1920x1080 TN 
  hide details  
Reply
Snowdevil
(16 items)
 
ASUS G750JM
(9 items)
 
 
CPUMotherboardGraphicsGraphics
[i7 4790K @ 4.4 GHz (1.186v)] [Asus Sabertooth Z97 Mark S] [nVidia Geforce GTX 1080] [nVidia Geforce GTX 1080] 
RAMHard DriveCoolingOS
[G.Skill 32GB DDR3 2133 MHz] [Crucial MX100 256GB] [Phanteks PH-TC12DX] [Win 10.1 Pro] 
MonitorMonitorKeyboardPower
[LG 29UM65 (2560x1080)] [QNIX Evo II LED (2560x1440)] [WASD v2 Tenkeyless] [NZXT Hale90 v2 ] 
CaseMouseMouse PadAudio
[ThermalTake GT10 Snow Edition] [Razer Mamba - Chroma] [Razer Kabuto] [Razer Man O' War] 
CPUMotherboardGraphicsRAM
i7 4770HQ Intel HM87 Express Chipset Geforce GTX 860M 8GB DDR3L 1600 MHz 
Hard DriveOptical DriveCoolingOS
Samsung SSD EVO DVD-RW Stock Windows 8.1 
Monitor
1920x1080 TN 
  hide details  
Reply
post #3 of 30
Thread Starter 
Quote:
Originally Posted by RagingCain View Post

Have you just tried raw conversion to integer base 16 and then back to Unicode characters to see if it is actually encoded as? I know nothing really of encoding/decoding. So I assumed you tried converting to plain text format.

If doing this through Windows / VSStudio then you have .NET Framework, there is an opensource class with a .ToString() built in.
http://www.docs.snmpsharpnet.com/docs-0-5-0/html/N_SnmpSharpNet.htm

Don't re-invent the wheel if you don't have to.

Thanks for your input smile.gif

I have kind of figured it out now. It was simply working off a TLV basis (Type, length, value). However new problem biggrin.gif


See OP I am reading data byte by byte. The array is an unsigned char; however whenever the value of the byte exceeds 127 I get garbled data (like 4627201 or something). I am not understanding why.

Under 127 the output is fine, correct.
Edited by mikeo01 - 4/20/14 at 6:06am
post #4 of 30
Quote:
Originally Posted by mikeo01 View Post

Quote:
Originally Posted by RagingCain View Post

Have you just tried raw conversion to integer base 16 and then back to Unicode characters to see if it is actually encoded as? I know nothing really of encoding/decoding. So I assumed you tried converting to plain text format.

If doing this through Windows / VSStudio then you have .NET Framework, there is an opensource class with a .ToString() built in.
http://www.docs.snmpsharpnet.com/docs-0-5-0/html/N_SnmpSharpNet.htm

Don't re-invent the wheel if you don't have to.

Thanks for your input smile.gif

I have kind of figured it out now. It was simply working off a TLV basis (Type, length, value). However new problem biggrin.gif


See OP I am reading data byte by byte. The array is an unsigned char; however whenever the value of the byte exceeds 127 I get garbled data (like 4627201 or something). I am not understanding why.

Under 127 the output is fine, correct.


Now I maybe able to help.

I think the values are either being stored as signed 8-bit or retrieved that way.

Signed 8-bit is stored in values from -127 to 127, including 0, makes a total of 256 possible values. Atypically this is the maximum one can store in an individual byte. Try to see if byte data type is actually unsigned not just what you are putting it in as, the dynamics of signed data can not just be casted to an unsigned variable as it is passed by value (guessing) giving you a very large positive number.

All this is, is a classic case of memory overflow without exception or error.
Edited by RagingCain - 4/20/14 at 6:32am
Snowdevil
(16 items)
 
ASUS G750JM
(9 items)
 
 
CPUMotherboardGraphicsGraphics
[i7 4790K @ 4.4 GHz (1.186v)] [Asus Sabertooth Z97 Mark S] [nVidia Geforce GTX 1080] [nVidia Geforce GTX 1080] 
RAMHard DriveCoolingOS
[G.Skill 32GB DDR3 2133 MHz] [Crucial MX100 256GB] [Phanteks PH-TC12DX] [Win 10.1 Pro] 
MonitorMonitorKeyboardPower
[LG 29UM65 (2560x1080)] [QNIX Evo II LED (2560x1440)] [WASD v2 Tenkeyless] [NZXT Hale90 v2 ] 
CaseMouseMouse PadAudio
[ThermalTake GT10 Snow Edition] [Razer Mamba - Chroma] [Razer Kabuto] [Razer Man O' War] 
CPUMotherboardGraphicsRAM
i7 4770HQ Intel HM87 Express Chipset Geforce GTX 860M 8GB DDR3L 1600 MHz 
Hard DriveOptical DriveCoolingOS
Samsung SSD EVO DVD-RW Stock Windows 8.1 
Monitor
1920x1080 TN 
  hide details  
Reply
Snowdevil
(16 items)
 
ASUS G750JM
(9 items)
 
 
CPUMotherboardGraphicsGraphics
[i7 4790K @ 4.4 GHz (1.186v)] [Asus Sabertooth Z97 Mark S] [nVidia Geforce GTX 1080] [nVidia Geforce GTX 1080] 
RAMHard DriveCoolingOS
[G.Skill 32GB DDR3 2133 MHz] [Crucial MX100 256GB] [Phanteks PH-TC12DX] [Win 10.1 Pro] 
MonitorMonitorKeyboardPower
[LG 29UM65 (2560x1080)] [QNIX Evo II LED (2560x1440)] [WASD v2 Tenkeyless] [NZXT Hale90 v2 ] 
CaseMouseMouse PadAudio
[ThermalTake GT10 Snow Edition] [Razer Mamba - Chroma] [Razer Kabuto] [Razer Man O' War] 
CPUMotherboardGraphicsRAM
i7 4770HQ Intel HM87 Express Chipset Geforce GTX 860M 8GB DDR3L 1600 MHz 
Hard DriveOptical DriveCoolingOS
Samsung SSD EVO DVD-RW Stock Windows 8.1 
Monitor
1920x1080 TN 
  hide details  
Reply
post #5 of 30
Thread Starter 
Quote:
Originally Posted by RagingCain View Post

Now I maybe able to help.

I think the values are either being stored as signed 8-bit or retrieved that way.

Signed 8-bit is stored in values from -127 to 127, including 0, makes a total of 256 possible values. Atypically this is the maximum one can store in an individual byte. Try to see if byte data type is actually unsigned not just what you are putting it in as, the dynamics of signed data can not just be casted to an unsigned variable as it is passed by value (guessing) giving you a very large positive number.

All this is, is a classic case of memory overflow without exception or error.

Ah thanks for your help. Maybe I am getting a bit confused with what I am doing.

Here's some code
Code:
char *BUFF = (char *)malloc(65536);
SRC=recvfrom(SNMP_ANALYSE,BUFF,65536,0,0,0);

Basically I have allocated memory for whatever I will receive and use RECVFROM() to retrieve them. SRC is basically the integer value of the total length of the packet.

... But RECVFROM() only accepts a char? A char can be signed or unsigned.

If I treat the variable BUFF as unsigned RECVFROM() complains it cannot convert "unsigned char to char *).


*BUFF is passed over and the data payload field is calculated. So I am left with the data payload field in *BUFF to process.


So, if I am getting a very large positive number (or -127 if I treat as signed) why am I able to correctly calculate the hexadecimal value of whatever is stored within that byte?
Edited by mikeo01 - 4/20/14 at 6:53am
post #6 of 30
Sounds like you are converting your signed char to a 32 bit signed int, then printing the int as if it were unsigned. Sign extension would occur during the conversion from char to int if your char is < 0.

http://en.wikipedia.org/wiki/Sign_extension

It's hard to tell what you're doing wrong without code, but you're probably just using/printing the values from BUFF incorrectly.
Edited by morethantoast - 4/20/14 at 7:25am
eyefinity
(22 items)
 
  
CPUMotherboardGraphicsGraphics
Intel Core i7 2600K EVGA E685 Radeon HD 6970 Radeon HD 6970 
RAMRAMHard DriveCooling
G.Skill SNIPER G.Skill SNIPER Crucial M4 128GB 6970 EK Water Block Acetal + Nickel 
CoolingCoolingCoolingOS
Swiftech MCR320-QP Radiator Swiftech MCP655 Pump EK Supreme HF CPU Block Windows 7 Ultimate 
MonitorMonitorMonitorKeyboard
Dell E228WFP Dell E228WFP Dell P2210 G510 
PowerCaseMouseMouse Pad
TX850 HAF932 BE G500 XTRAC PADS Ripper 
Audio
G930 
  hide details  
Reply
eyefinity
(22 items)
 
  
CPUMotherboardGraphicsGraphics
Intel Core i7 2600K EVGA E685 Radeon HD 6970 Radeon HD 6970 
RAMRAMHard DriveCooling
G.Skill SNIPER G.Skill SNIPER Crucial M4 128GB 6970 EK Water Block Acetal + Nickel 
CoolingCoolingCoolingOS
Swiftech MCR320-QP Radiator Swiftech MCP655 Pump EK Supreme HF CPU Block Windows 7 Ultimate 
MonitorMonitorMonitorKeyboard
Dell E228WFP Dell E228WFP Dell P2210 G510 
PowerCaseMouseMouse Pad
TX850 HAF932 BE G500 XTRAC PADS Ripper 
Audio
G930 
  hide details  
Reply
post #7 of 30
Thread Starter 
Quote:
Originally Posted by morethantoast View Post

Sounds like you are converting your signed char to a 32 bit signed int, then printing the int as if it were unsigned. Sign extension would occur during the conversion from char to int if your char is < 0.

http://en.wikipedia.org/wiki/Sign_extension

It's hard to tell what you're doing wrong without code, but I imagine you might be doing something like this
Code:
printf("%u", BUFF[i])

which would cause the results you are seeing if BUFF[eye] < 0

I'll print a part of my code out.
Code:
CHARACTER=DATA[X];

                HEX=(DATA[X]>>4&0x0f); // Bitwise shift operator >> by 4 bits and AND, mask bottom most nibble (4-bits) to calculate the hex value
                
                if(F_TYPE==3){L_NIBBLE=HEX;}
                if(HEX<10){C='0'+HEX;} // If decimal value is under 10 add the HEX value. If over ten work letter (divide by 10, take reminder and add onto C = letter result)
                else{C='A'+(HEX%10);}


                SNMP_LOG << (unsigned char) C; // Add the hex value to the text file

                HEX=(DATA[X]&0x0f); // Bitwise AND to calculate bottom most nibble (4-bits)

                if(HEX<10){C='0'+HEX;} // Calculate HEX
                else{C='A'+(HEX%10);}
                if(F_TYPE==3){H_NIBBLE=HEX;}

I am putting whatever is in my byte array (DATA which was *BUFF) into CHARACTER (char).

I am then working out the hexadecimal values of each nibble using bitwise operations. The hexadecimal values are outputted to a text file.

Further down in my code the ASCII values of each byte are printed out (if it's a STRING or OCTET STRING type).

Ignore HOP, TLV, SIGNAL and SEQ variables they are used to determine SNMP values. Essentially L_NIBBLE and H_NIBBLE are the two HEX variables above. They are the left overs of the bitwise operators.
Code:
if(HOP==0)
                                        {
                                        std::ostringstream TMP;
                                        TMP << (unsigned int) L_NIBBLE << (unsigned int) H_NIBBLE
                                        std::istringstream CONV(TMP.str()); // Combine Integers
                                        CONV >> LEN;
                                        if((LEN>=80) || (LEN==0) || (LEN==00)){TLV=0;TEMP=TEMP+LEN;SIGNAL=1;HOP=0;} // SIGNAL = that there is more data to follow                                    
                                        else
                                        {
                                                if(SEQ==1)
                                                {
                                                SNMP_MESSAGE << "Field Length (Bytes): ";
                                                if(SIGNAL==0){SNMP_MESSAGE << LEN << "\n";}else{SNMP_MESSAGE << TEMP << "\n";}
                                                SEQ=0;
                                                TLV=2;
                                                SIGNAL=0;
                                                HOP=0;
                                                }
                                                else{HOP=1;TLV=0;}
                                        }
                                        }
                                        else{TLV=1;}
                                } // Finish LENGTH

What I was originally doing was putting the nibbles back together, which is wrong (str() is wrong), as the combinations would only allow up to 15 (because I split the byte up).

LEN and TEMP are variables to determine the field lengths. If the total byte value was greater than 127 I'd put this into TEMP to work out the next data to follow.


So I figured I could just use:
Code:
TEXT_FILE << (unsigned int) CHARACTER

To get the value of the CHARACTER variable earlier.
Edited by mikeo01 - 4/20/14 at 7:41am
post #8 of 30
Thread Starter 
I'll post again as the above is a bit messy tongue.gif

I have stored the original data into CHARACTER before messing with the original data.

Essentially I am making a bitwise shift operation on the original DATA to determine their lower and upper nibble values (to work out the hexadecimal of them).

Later on in the code I use CHARACTER to print out the ASCII values (in regards to the data type).


Within SNMP I need to know the field length to know how far to print out data, so when the field starts and ends basically.


The str() is incorrect as I realised; I have split the data up so the bit pattern won't be correct. So I thought if I use CHARACTER and output it as an unsigned int I'd get the original byte values.

I.E. Hexadecimal and ASCII =

30 82 00 A3 02 01 00 04 0C 4D 4F 4E 49 54 4F 52 (in ASCII 0........MONITOR )

Decimal =

48 4294967170 0 4294967203 2 1 0 4 12 77 79 78 73 84 79 82


These decimal values are the original from the hexadecimal above. Every value is correct except for the ones above 127 (i.e., hex 82, A3 which are 130 and 163 in decimal)
post #9 of 30
It has been a while since my computer science data type class. I will go through your code later in more detail, but at first glance, Char is 2 bytes. That explains the huge number size.

Memory is using/getting this from Char:
00001111 00000000 (and it may not be 00000000) its going to read what is available in the next byte over, and obviously it is not being initialized to zero, but being read as is.

Instead of:
00001111

And if this is .NET Framework, the assembly/compiler is auto-fixing this, due to side-effect/autocasting. Explicit casting is going to be necessary to prevent this.

Your data is being converted from Integer 8 bit, to Int 32.

Memory reference sizes:
http://msdn.microsoft.com/en-us/library/eahchzkf.aspx
Edited by RagingCain - 4/20/14 at 9:42am
Snowdevil
(16 items)
 
ASUS G750JM
(9 items)
 
 
CPUMotherboardGraphicsGraphics
[i7 4790K @ 4.4 GHz (1.186v)] [Asus Sabertooth Z97 Mark S] [nVidia Geforce GTX 1080] [nVidia Geforce GTX 1080] 
RAMHard DriveCoolingOS
[G.Skill 32GB DDR3 2133 MHz] [Crucial MX100 256GB] [Phanteks PH-TC12DX] [Win 10.1 Pro] 
MonitorMonitorKeyboardPower
[LG 29UM65 (2560x1080)] [QNIX Evo II LED (2560x1440)] [WASD v2 Tenkeyless] [NZXT Hale90 v2 ] 
CaseMouseMouse PadAudio
[ThermalTake GT10 Snow Edition] [Razer Mamba - Chroma] [Razer Kabuto] [Razer Man O' War] 
CPUMotherboardGraphicsRAM
i7 4770HQ Intel HM87 Express Chipset Geforce GTX 860M 8GB DDR3L 1600 MHz 
Hard DriveOptical DriveCoolingOS
Samsung SSD EVO DVD-RW Stock Windows 8.1 
Monitor
1920x1080 TN 
  hide details  
Reply
Snowdevil
(16 items)
 
ASUS G750JM
(9 items)
 
 
CPUMotherboardGraphicsGraphics
[i7 4790K @ 4.4 GHz (1.186v)] [Asus Sabertooth Z97 Mark S] [nVidia Geforce GTX 1080] [nVidia Geforce GTX 1080] 
RAMHard DriveCoolingOS
[G.Skill 32GB DDR3 2133 MHz] [Crucial MX100 256GB] [Phanteks PH-TC12DX] [Win 10.1 Pro] 
MonitorMonitorKeyboardPower
[LG 29UM65 (2560x1080)] [QNIX Evo II LED (2560x1440)] [WASD v2 Tenkeyless] [NZXT Hale90 v2 ] 
CaseMouseMouse PadAudio
[ThermalTake GT10 Snow Edition] [Razer Mamba - Chroma] [Razer Kabuto] [Razer Man O' War] 
CPUMotherboardGraphicsRAM
i7 4770HQ Intel HM87 Express Chipset Geforce GTX 860M 8GB DDR3L 1600 MHz 
Hard DriveOptical DriveCoolingOS
Samsung SSD EVO DVD-RW Stock Windows 8.1 
Monitor
1920x1080 TN 
  hide details  
Reply
post #10 of 30
Thread Starter 
Quote:
Originally Posted by RagingCain View Post

It has been a while since my computer science data type class. I will go through your code later in more detail, but at first glance, Char is 2 bytes. That explains the huge number size.

Memory is using/getting this from Char:
00001111 00000000 (and it may not be 00000000) its going to read what is available in the next byte over, and obviously it is not being initialized to zero, but being read as is.

Instead of:
00001111

And if this is .NET Framework, the assembly/compiler is auto-fixing this, due to side-effect/autocasting. Explicit casting is going to be necessary to prevent this.

Your data is being converted from Integer 8 bit, to Int 32.

Memory reference sizes:
http://msdn.microsoft.com/en-us/library/eahchzkf.aspx

Char is two bytes? Thanks for clearing that up! I didn't even know that. Something so simple yet I completely missed that; I got lucky with my program design.

Well for reference, a user from a different forum suggested I use << (unsigned int) (DATA&0x0f).

So that is completely odd. So essentially that is wasted space? I mean if everything else is working and displaying the correct data as it should, does that mean the second byte in the char array is in fact 00000000?

So half of my char array is 0? As I am working with hex values I have used (&0x0f) and (>>4&0x0f); without actually realising what my code is doing? Although that works I am simply keeping the initial byte and "masking" the first 4 bits then shifting over and doing the same?


Which makes me wonder, is there anything else in the second byte when I shift through my data?

Thanks for your help thumb.gif +rep
Edited by mikeo01 - 4/20/14 at 10:48am
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Coding and Programming
Overclock.net › Forums › Software, Programming and Coding › Coding and Programming › Data Payload Decoding - Issues with Byte Reading