Overclock.net banner
1 - 10 of 10 Posts

·
Old dog, old tricks
Joined
·
9,756 Posts
Discussion Starter · #1 ·
Hi guys. I need a tip on the (de)serialization on C#.

I got a class with 2xDictionary(Index, Word)

Index contains an int[] of 1-4 ints,
Word contains 3xString

With 100,000 entries in each Dictionary in the class, 13Mb, deserialization takes me about 5 minutes. First of all, is this normal?
I'm making a dictionary and I don't think lots of people will hang around 10 min for a simple dictionary to load. So please, any advise on performance boost is precious. (The CPU is Celeron 64 1.9G)
 

·
Old dog, old tricks
Joined
·
9,756 Posts
Discussion Starter · #3 ·
Binary, but tried both, same timings. XML a bit slower
 

·
Old dog, old tricks
Joined
·
9,756 Posts
Discussion Starter · #5 ·
to be honest, I don't know what a profiler is.

Here's the sample

[Serializable]
class Language
{
public String name;

public SortedDictionary<Index, IWord> indexData;
public SortedDictionary<IWord, Index> wordData;
public List<Char> charSet;
...Constructor...
}

[Serializable]
class Index : IComparable
{
public List<int> indexes;
public Index(int[] indexes1)
{
indexes = new List<int>();
indexes.AddRange(indexes1);
}
...method CompareTo()...
}

[Serializable]
class Word : IWord, IComparable
{
public String word;
public String[] types; // verb, noun, adjective, pronoun, preposition, adverb, conjunction and interjection.
public String description;
public List<String> notes;
public Language lang;
...Constructor & Get-Set...
}

class Serializer
{
public static Language Load(String name)
{
Stream stream = null;
Language lang = null;
try
{
IFormatter formatter = new BinaryFormatter();
stream = new FileStream(Application.StartupPath + "\\\\Languages\\\\"+name + ".lng", FileMode.Open, FileAccess.Read, FileShare.None);
stream.Position = 0;
lang = (Language)formatter.Deserialize(stream);
}
catch
{
// do nothing, just ignore any possible errors
}
finally
{
if (stream != null)
stream.Close();
}
return lang;
}
}

I have no idea what might be wrong. In the test case the 2 Dictionaries in Langiage have 100,000 entries. Is it too much, therefore slowing the whole thing down?

EDIT: Also tried the Index with arrays int[], same result. In the test case, each holds 2 ints.
 

·
Old dog, old tricks
Joined
·
9,756 Posts
Discussion Starter · #6 ·
Ok, just tried with 10,000 entries instead of 100,000 and it loads Immediately! In matters of half second. The problem is- on runtime, my collections are more likely to have 100,000 entries than 10,000. And also why is it not linear? what could be the cause?
10,000 - 1.3s Total
100,000 - 5+ min Each
 

·
Registered
Joined
·
6,845 Posts
This isn't particularly helpful, but if I was dealing with that amount of data, I'd be looking to get it into a SQL table (maybe SQLite, if C# has bindings for it - I'm sure it must have!).

That said, there should be another way of storing the data that's more efficient. Most databases are quick due to indexing, so you may be able to work something in that way? Maybe recode the storage as a C function/pointer, as I'm not sure if C# is slower to run?
 

·
Old dog, old tricks
Joined
·
9,756 Posts
Discussion Starter · #8 ·
Quote:


Originally Posted by chemicalfan
View Post

This isn't particularly helpful, but if I was dealing with that amount of data, I'd be looking to get it into a SQL table (maybe SQLite, if C# has bindings for it - I'm sure it must have!).

That said, there should be another way of storing the data that's more efficient. Most databases are quick due to indexing, so you may be able to work something in that way? Maybe recode the storage as a C function/pointer, as I'm not sure if C# is slower to run?

That's right, I would be using SQL Compact and I will but in a later release. The thing is, I didn't have the time to implement any DB, also I don't have any experience with integrated DB, I've only used SQL 2008 Express.

Now, just for info for anybody that might be interested, here is a DataStruct performance comparisson, I couldn't find any on google, so I did a quick benchmark myself. Here's the result: (All values are in millisec.)

STRING TEST
String Array of 100,000 constructed in 0
String List constructed in 0
String Linked List constructed in 0
String Dictionary constructed in 0
Sorted String Dictionary constructed in 0
Sorted String List constructed in 0

Writing to Array..
Writing 100,000 entries to Array took 31
Writing to List..
Writing 100,000 entries to List took 46
Writing to Linked List..
Writing 100,000 entries to Linked List took 46
Writing to Dictionary..
Writing 100,000 key/value pairs to Dictionary took 109
Writing to Sorted Dictionary..
Writing 100,000 key/value pairs to Sorted Dictionary took 951
Writing to Sorted List..
Writing 100,000 key/value pairs to Sorted List took 762

Locating 98312 in Array..
Parsing Array took 0
Locating 98312 in List..
Parsing List took 15
Locating 98312 in Linked List..
Parsing Linked List took 0
Locating 98312 in Dictionary..
Parsing Dictionary took 0
Locating 98312 in Sorted Dictionary..
Parsing Sorted Dictionary took 46
Locating 98312 in Sorted Dictionary (Binary Search)..
Parsing Sorted Dictionary (Binary Search) took 15
Locating 98312 in Sorted List (Binary Search)..
Parsing Sorted List (Binary Search) took 0

INT TEST
Int Array of 100,000 constructed in 0
Int List constructed in 0
Int Linked List constructed in 15
Int Dictionary constructed in 0
Sorted Int Dictionary constructed in 0
Sorted Int List constructed in 0

Writing to Array..
Writing 100,000 entries to Array took 0
Writing to List..
Writing 100,000 entries to List took 0
Writing to Linked List..
Writing 100,000 entries to Linked List took 31
Writing to Dictionary..
Writing 100,000 key/value pairs to Dictionary took 15
Writing to Sorted Dictionary..
Writing 100,000 key/value pairs to Sorted Dictionary took 171
Writing to Sorted List..
Writing 100,000 key/value pairs to Sorted List took 31

Locating 98312 in Array..
Parsing Array took 0
Locating 98312 in List..
Parsing List took 0
Locating 98312 in Linked List..
Parsing Linked List took 15
Locating 98312 in Dictionary..
Parsing Dictionary took 0
Locating 98312 in Sorted Dictionary..
Parsing Sorted Dictionary took 15
Locating 98312 in Sorted Dictionary (Binary Search)..
Parsing Sorted Dictionary (Binary Search) took 46
Locating 98312 in Sorted List (Binary Search)..
Parsing Sorted List (Binary Search) took 0

OBJECT TEST
Int[] List constructed in 0
Writing..
Wrote 100,000 x 3 entries in 15
Locating 283992..
Located in 0
 

·
Old dog, old tricks
Joined
·
9,756 Posts
Discussion Starter · #10 ·
Ok, I just finished developing an alternative of the Sorted Dictionary, now available for free download on http://homedimension.webs.com .

Documentation will soon be available via http://homedimension.webs.com .

Features:
-Ability to add Multiple values per Key
-Ability to function as an ordinary List
-Can switch between sorted and unsorted
-After ~1000 string tests- 5 times faster Write, up to 2 times faster parsing, up to 3 times faster Binary Search
(for now you have to manually .Sort() after you adding/removing session)

I'm working on the documentation, everybody who can figure out how to use it on his own- go on, happy coding
.
 
1 - 10 of 10 Posts
This is an older thread, you may not receive a response, and could be reviving an old thread. Please consider creating a new thread.
Top