Overclock.net › Forums › Software, Programming and Coding › Coding and Programming › How can I retrieve SPF records for parsing?
New Posts  All Forums:Forum Nav:

How can I retrieve SPF records for parsing?

post #1 of 18
Thread Starter 
Hello Overclock.net!

I am willing to get into whatever programming language best does this JavaScript (web scripting not programming I KNOW!), C++ (real programming language...), python, whatever it may be!

My problem is I need to analyze over a thousand SPF records, I would like to simply get a .txt document (or equivalent for moderate level programming knowledge to parse) with each line having 1 SPF record.

I have NEVER found a way to pull HTML or data from the internet into a program, I would be fancinated to know how I can do this, because parsing internet data would be very useful.

WHAT I NEED TO DO:

1. from a list of about 1,000 different domains (in format example.com) type items I need to retrieve over 1,000 SPF records.

2. I need to identify a string within the records after I have the list of records: include:example.com, ?include:example.com (invalid for SPF syntax, but I need to know the invalid records too!)

3. I need a list of the remainder WITHOUT this record, and a list of the mal-formed records.

If any of you have anything you may know about the things I am asking, I would be extremely grateful to hear what you have to say! Even if it's one word I can google, please let me know!

thumb.gif Thank you all!
Ground Zero
(7 items)
 
  
CPUGraphicsRAMHard Drive
i5 Quad Core Intel GeForce GTX 780 8 GIG 256 GB SSD 
OSPowerCase
Windows 8.1  750W Seasonic  Corsair Carbide 200R 
  hide details  
Reply
Ground Zero
(7 items)
 
  
CPUGraphicsRAMHard Drive
i5 Quad Core Intel GeForce GTX 780 8 GIG 256 GB SSD 
OSPowerCase
Windows 8.1  750W Seasonic  Corsair Carbide 200R 
  hide details  
Reply
post #2 of 18
Thread Starter 
Please help me research this! I am googling away, but this is very difficult for me to understand how to make this happen!

http://stackoverflow.com/questions/11846705/get-spf-records-from-a-domain

[Update] The ARSoft.Tools.Net project looks interesting in this one above

https://en.wikipedia.org/wiki/Sender_Policy_Framework

http://stackoverflow.com/questions/4082935/using-dig-to-search-for-spf-records

https://www.madboa.com/geek/dig/

According to Wikipedia SPF records are used to... "mail exchangers to check that incoming mail" ...how can I mimic what these mail exchangers are doing?
Edited by thethispointer - 2/27/16 at 12:07pm
Ground Zero
(7 items)
 
  
CPUGraphicsRAMHard Drive
i5 Quad Core Intel GeForce GTX 780 8 GIG 256 GB SSD 
OSPowerCase
Windows 8.1  750W Seasonic  Corsair Carbide 200R 
  hide details  
Reply
Ground Zero
(7 items)
 
  
CPUGraphicsRAMHard Drive
i5 Quad Core Intel GeForce GTX 780 8 GIG 256 GB SSD 
OSPowerCase
Windows 8.1  750W Seasonic  Corsair Carbide 200R 
  hide details  
Reply
post #3 of 18
So you need to pull the SPF DNS records into a text file?

Bash is a wonderful method of doing that.
You can do this single command to look for TXT records.
Quote:

root@digi [~]# dig txt +short google.com
"v=spf1 include:_spf.google.com ~all"

Then you can use a list of domains to for loop through them and pull the same data.
Like this.
Quote:

root@digi [~]# vim domainlist.txt
root@digi [~]# for i in `cat domainlist.txt`; do dig txt +short $i; done
"v=spf1 include:_spf.google.com ~all"
"v=spf1 redirect=_spf.mail.yahoo.com"
"v=spf1 include:Spf-a.hotmail.com include:Spf-b.hotmail.com include:Spf-c.hotmail.com include:Spf-d.hotmail.com include:_spf-ssg-a.microsoft.com include:Spf.protection.outlook.com ~all"

I just pulled google, yahoo, and msn TXT records.
That command can be piped into a file to generate a list of those items.

And just to make sure you only get SPF records, you can do this
Quote:
root@digi [~]# for i in `cat domainlist.txt`; do dig txt +short $i; done |grep "v=spf"
"v=spf1 include:_spf.google.com ~all"
"v=spf1 redirect=_spf.mail.yahoo.com"
"v=spf1 include:Spf-a.hotmail.com include:Spf-b.hotmail.com include:Spf-c.hotmail.com include:Spf-d.hotmail.com include:_spf-ssg-a.microsoft.com include:Spf.protection.outlook.com ~all"
root@digi [~]#

PS:

This is a list of the top 10k domain names from some guys github.

https://github.com/opendns/public-domain-lists/blob/master/opendns-top-domains.txt
post #4 of 18
Thread Starter 
Wow thrasherht Thank you! thumb.gifbiggrin.gif REP+

...Bash is the GNU Project's shell... I don't yet know what I shell is. https://en.wikipedia.org/wiki/Bash_(Unix_shell)

I have taken C++ courses up to data structures. I have dabbled in other languages. This is FYI on my background.

This is probably perfect! Help me understand more. What OS are you using? Are you using command line or GUI? I am a windows guy through and through, but I am VERY literate and can pick up off youtube pretty well. What language did you just write in? Was that not a language but only command line prompts? Finally, I am familiar with GUI OSs being young (24) do you know if there's buzz words I can google to get a crash course on this shell stuff?
Quote:
Originally Posted by thrasherht View Post

So you need to pull the SPF DNS records into a text file?

Bash is a wonderful method of doing that.
You can do this single command to look for TXT records.
Then you can use a list of domains to for loop through them and pull the same data.
Like this.
I just pulled google, yahoo, and msn TXT records.
That command can be piped into a file to generate a list of those items.

And just to make sure you only get SPF records, you can do this
Ground Zero
(7 items)
 
  
CPUGraphicsRAMHard Drive
i5 Quad Core Intel GeForce GTX 780 8 GIG 256 GB SSD 
OSPowerCase
Windows 8.1  750W Seasonic  Corsair Carbide 200R 
  hide details  
Reply
Ground Zero
(7 items)
 
  
CPUGraphicsRAMHard Drive
i5 Quad Core Intel GeForce GTX 780 8 GIG 256 GB SSD 
OSPowerCase
Windows 8.1  750W Seasonic  Corsair Carbide 200R 
  hide details  
Reply
post #5 of 18
Thread Starter 
Very resourceful! If you don't mind me asking, what do you do for a living? What education have you been fortunate to have? I assume programming. thumb.gif
Quote:
Originally Posted by thrasherht View Post

PS:

This is a list of the top 10k domain names from some guys github.

https://github.com/opendns/public-domain-lists/blob/master/opendns-top-domains.txt
Ground Zero
(7 items)
 
  
CPUGraphicsRAMHard Drive
i5 Quad Core Intel GeForce GTX 780 8 GIG 256 GB SSD 
OSPowerCase
Windows 8.1  750W Seasonic  Corsair Carbide 200R 
  hide details  
Reply
Ground Zero
(7 items)
 
  
CPUGraphicsRAMHard Drive
i5 Quad Core Intel GeForce GTX 780 8 GIG 256 GB SSD 
OSPowerCase
Windows 8.1  750W Seasonic  Corsair Carbide 200R 
  hide details  
Reply
post #6 of 18
All of the stuff you saw in my post was from a Linux SSH terminal, (Command line)
Bash is just what some people call command line, as it is a specific type of command line in linux.

I wouldn't know how to do what you are attempting via C++ or other programming languages.
I am just a linux administrator. So I do that type of stuff everyday at work.

Basically what I showed you a simple form of bash scripting.
The difference is I ran the commands directly rather then through a script.
When using a script you basically put all those same command into a file, and then execute the file, which in turn executes the commands you put in it.

This site is great for linux related info, and that page has lots of good links.
http://www.cyberciti.biz/open-source/learning-bash-scripting-for-beginners/
post #7 of 18
Thread Starter 
This is gold! Thrasherht programming was the word I chose in ignorance, the requirement is not that I program this. I only need the results. Boom!

As a linux admin do you make sure a server is up and running without issue?
Quote:
Originally Posted by thrasherht View Post

All of the stuff you saw in my post was from a Linux SSH terminal, (Command line)
Bash is just what some people call command line, as it is a specific type of command line in linux.

I wouldn't know how to do what you are attempting via C++ or other programming languages.
I am just a linux administrator. So I do that type of stuff everyday at work.

Basically what I showed you a simple form of bash scripting.
The difference is I ran the commands directly rather then through a script.
When using a script you basically put all those same command into a file, and then execute the file, which in turn executes the commands you put in it.

This site is great for linux related info, and that page has lots of good links.
http://www.cyberciti.biz/open-source/learning-bash-scripting-for-beginners/
Ground Zero
(7 items)
 
  
CPUGraphicsRAMHard Drive
i5 Quad Core Intel GeForce GTX 780 8 GIG 256 GB SSD 
OSPowerCase
Windows 8.1  750W Seasonic  Corsair Carbide 200R 
  hide details  
Reply
Ground Zero
(7 items)
 
  
CPUGraphicsRAMHard Drive
i5 Quad Core Intel GeForce GTX 780 8 GIG 256 GB SSD 
OSPowerCase
Windows 8.1  750W Seasonic  Corsair Carbide 200R 
  hide details  
Reply
post #8 of 18
My education is basically a highschool diploma, a small amount of college in Information Technology.
Then the rest is either self taught, or I learned on the job at my current job.

I work as a server administrator for liquidweb, which is a hosting company.
I recently went through the Redhat Certified System Administrator (RHCSA) class, and got certified for Redhat Enterprise Linux 7

I am only 25, so basically same age as you.
Bash is a glorious tool, and can be so powerful.
post #9 of 18
Thread Starter 
Indeed! I went to UNL (Lincoln, Nebraska) for a while and it wasn't my thing. Came back to Omaha and been working on getting a General Information Technology Associates of Science (GITAS) and some CompTIA certifications to kick it off: IT Fundamentals (replaced strata), A+, Network+, Security+.

Great info!
Quote:
Originally Posted by thrasherht View Post

My education is basically a highschool diploma, a small amount of college in Information Technology.
Then the rest is either self taught, or I learned on the job at my current job.

I work as a server administrator for liquidweb, which is a hosting company.
I recently went through the Redhat Certified System Administrator (RHCSA) class, and got certified for Redhat Enterprise Linux 7
Ground Zero
(7 items)
 
  
CPUGraphicsRAMHard Drive
i5 Quad Core Intel GeForce GTX 780 8 GIG 256 GB SSD 
OSPowerCase
Windows 8.1  750W Seasonic  Corsair Carbide 200R 
  hide details  
Reply
Ground Zero
(7 items)
 
  
CPUGraphicsRAMHard Drive
i5 Quad Core Intel GeForce GTX 780 8 GIG 256 GB SSD 
OSPowerCase
Windows 8.1  750W Seasonic  Corsair Carbide 200R 
  hide details  
Reply
post #10 of 18
Here is an example of what you can do with bash.
I have a parent server in the rack at work. It runs a software called proxmox, which just lets me kick virtual instances onto that parent server.

Well that server sits behind a single IP, so giving each instance its own IP isn't an option.
So I setup NAT through Iptables (basic firewall software) and wrote a script to configure the nat every time a new instance is added or removed.

Here is that script.
https://github.com/thrasherht/openvz-port-forward/blob/master/iptables-setup.sh

This is how it works when run.

Gives you a menu
Quote:
~~~~~~~~~~~~~~~~~~~~~
M A I N - M E N U
~~~~~~~~~~~~~~~~~~~~~
1. Forward a port to an instance
2. Restore iptables configuration from file
3. Display live IPtables configuration
4. Exit
Enter choice [ 1 - 3]

Then asks for the instance IP, and the ports on the inside and outside to forward.
Quote:
~~~~~~~~~~~~~~~~~~~~~
M A I N - M E N U
~~~~~~~~~~~~~~~~~~~~~
1. Forward a port to an instance
2. Restore iptables configuration from file
3. Display live IPtables configuration
4. Exit
Enter choice [ 1 - 3] 1
Enter private instance IP: 192.168.2.150
Enter destination port: 2000
Enter external port to be forwarded: 2000
Saving Backup of iptables rules
Rules saved to /backup/iptables-backups/iptables.rules.2016-02-27.15:25
Adding rule to live iptables configuration
Saving new configuration to persistant file
Press [Enter] key to continue...
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Coding and Programming
Overclock.net › Forums › Software, Programming and Coding › Coding and Programming › How can I retrieve SPF records for parsing?