Writing to files faster in Python -

i have been mining tweets. main problem have been facing - have encode tweets utf-8 , write them file.

my current method:

def on_data(self,data):     f=open('new','w')     dict1=json.loads(data)     val=dict1["text"]     val= codecs.encode(val,"utf-8","ignore")     var.x+=1     f.write(str(var.x)+"\t"+val+"\n")     return true 

any way speed process?

you're not closing file, means have wait until python figures out can safely closed when file handle isn't in use more.

assuming want create new empty file every time on_data() called, can use with statement have python close file when with block exited:

def on_data(self, data):     dict1 = json.loads(data)     val = dict1["text"]     val = codecs.encode(val,"utf-8", "ignore")     var.x+=1     open('new', 'w') f:         f.write(str(var.x) + "\t" + val + "\n")     return true