Tuesday, January 28, 2014

Comparision of different concurrency models: Actors, CSP, Disruptor and Threads

In order to make use of modern multi core/socket hardware, Java offers threads and locks for concurrent programming. It is well known this model suffers of a number of problems:
  • Deadlocks
  • Bad scaling due to "over-synchronization" or contention on common resources
  • Errors caused by invalid synchronization are timing dependent and hard to reproduce. It is common they appear under special conditions (load, hardware) in production.
  • As a software project grows, it gets hard to safely modify the application. A programmer needs global awareness of an applications control flow and data access patterns in order to change the software without introducing errors.
  • The subtleties of the Java Memory Model are not well known to most Java programmers. Even if JMM is known, incorrect programming isn't obvious and will fail rarely. It can be a nightmare to spot them from production logs of large applications.
So I am looking for alternative models to express concurrency. Therefore a simple benchmark of the most popular "alternative" concurrency models is done: Actors, CSP and Disruptor.



Actors


Real Life Analogy: People cooperating by sending mails to each other.

An Actor is like an object instance executed by a single thread. Instead of direct calls to methods, messages are put into the Actors "mailbox" (~queue). The actor single threaded reads and processes messages from the queue sequentially (with exceptions).
Internal state is exposed/shared by passing messages (with copied state) to other Actors.







CSP (Communicating Sequential Processes)


Real Life Analogy: People phoning each other.

Though a different terminology is used, CSP systems can be seen as a special actor system having bounded mailboxes (queues) of size 0.

So if one process (~Actor) wants to pass a message to another process, the caller is blocked until the receiver accepts the message. Alternatively to being blocked, a CSP-process can choose to do other things e.g. check for incoming messages (this introduces  non determinism to the order of outgoing messages). A receiver cannot accept incoming messages if he is occupied with processing a prior one.





Disruptor


The disruptor data structure came up some years ago and was invented and pioneered by Martin Thompson, Mike Barker, and Dave Farley at LMAX exchange.

Real Life Analogy: Assembly line




The disruptor is a bounded queue (implemented by a ring buffer) where producers add to the head of the queue (if slots are available, else the producer is blocked).
Consumers access the queue at different points, so each consumer has its own read position (cursor) inside the queue. Sequencing is used to manage queue access and processing order.



Thread + locking of shared data


contention
Ok, everybody knows this. Its Java's default to model concurrent execution. Actually these are the primitives (+CAS) used to build higher level concurrent models like those described above.











Conclusion


While the CSP/Actor-Model realizes inter-thread communication by passing/copying thread-local data to some queue or process, the Disruptor keeps data in place and assures only one thread (consumer or producer) is owner of a data item (=queue slot) at a time.
This model seems to fit existing CPU, memory and VM architectures better resulting in high throughput and superior performance. It avoids high allocation rates and reduces the probability of cache misses. Additionally it implicitly balances processing speed of interdependent consumers without growing queues somewhere.
There is no silver bullet for concurrent programming, one still has to plan carefully and needs to know what's going on.

While I am writing this, I realize each of these patterns has its set of best-fit problem domains, so my initial intention using a single benchmark to compare performance of different concurrency models might be somewhat off :-).


 

The Benchmark Application


 

Update: Jason Koch implemented a custom executor performing significantly better than the stock JDK executors. See his blog post.

The benchmark approximates the value of PI concurrently. Therefore N jobs are created, and the result of each job must be added to finally get an approximation of PI. For maximum performance one would create one large job for each Core of the CPU used. With this setup all solutions roughly would perform the same as there is no significant concurrent access and scheduling overhead.

However if we compute PI by creating 1 million jobs, each computing a 0..100-loop slice of PI, we get an impression of:
  • cost of accessing shared data when the result of each job is added to one single result value
  • overhead created by the concurrency control mechanics (e.g. sending messages/scheduling Runnables to a thread pool, CAS, Locks, volatile r/w ..).
The test is run with 2 configurations
  • 100 iterations per pi-slice-job, 1,000,000 jobs
  • 1,000 iterations per pi-slice-job, 100,000 jobs
As hardware I use
  • AMD Opteron 6274 Dual Socket 2,2 Ghz. Each socket has 8 cores + 8 Hardware Threads (so overall 16 cores + 16 HW Threads)
  • Intel XEON@3Ghz dual socket six core (12 cores + Hyperthreading turned off). (2011 release)
I never use more threads as there are "real" cores.



Stop the blabber, gimme results ! 


See bottom of this post for the benchmark source.
  • Threads - somewhat naive thread based implementation of the Pi computation. Enough effort invested it is possible to match any of the other results of course. At the core of the VM, threads, locks (+atomic, volatile r/w + CAS) are the only concurrent primitives. However there is no point in creating an ad-hoc copy of the Disruptor or an Actor system in order to compare concurrency approaches.
  • Akka - a popular Actor implementation on the VM. The benchmark has been reviewed and revised (especially the ActorSystem configuration can make a big difference) by the Akka crew. Threads are scheduled using Java 7's fork join pool. Actually the Pi computation is one of Akka's tutorial examples. 
  • Abstraktor - my experimental Actor/CSP implementation. It's using short bounded queues (so leans more to the CSP side) and avoids deadlocks by maintaining 2 queues per Actor (in and out). If the out-queue is blocked, it just reads from the in-queue.
    I am using Nitsan Wakarts excellent MPSC queue implementation (check his blog or github jaq-in-a-box) and that's the major reason it shows kind of competitive performance+scaling.
    I use this to get a rough baseline for comparision and experiment with different flavours of Actors/CSP. Probably the only thing one can do with it is to run the Pi bench ;-).
    Update: The experimental version benchmarked here has been consolidated + improved. You can find it ohttps://github.com/RuedigerMoeller/kontraktor
  • Disruptor - my naive approach implementing the benchmark based on the Disruptor 3.2. It turned out that I used a not-yet-polished utility class, however I keep this benchmark just to illustrate how smallish differences in implementation may have big consequences.
  • Disruptor2 - As Michael Barker would have implemented it (Thanks :-) ). Its actually more than twice as fast for the 1 million test as the closest runner up.
Intel (Xeon 2 socket each 6 cores, Hyperthreading off)

Well, it seems with 100k jobs of 1000 iterations, the benchmark is dominated by computation, not concurrency control. Therefore I retry with 1 million jobs each computing a 100 iteration slice.

Ok, we probably can see differences here :-)



AMD Opteron 6274 Dual Socket 2,2 Ghz (=16 real cores, 16 HW Threads)


Again the 1 million tiny-job variant spreads the difference amongst the approaches (and their implementation):


Note there is like 5% run to run jitter (GC and stuff), however that does not change the big picture.

Last but not least: Best results per CPU architecture per lib:




Discussion of Results


We see that scaling behaviour can be significantly different depending on the hardware platform used.

Akka loses a lot of CPU time doing GC and allocation. If I modify Abstraktor to use unbounded Concurrent Linked Queue (Akka uses those), it performs similar to Akka and builds up temporary queues >100k elements. This is an inherent issue of the Actor model. By leaning towards CSP (use very short bounded blocking queues with size <100), performance also suffers, as threads are blocked/spinning for input too often. However with a queue size of 5000 elements, things work out pretty well (introducing other problems like deadlocks in case of cyclic Actor graphs).
The Disruptor library is very well implemented, so a good part of the performance advantage could be attributed to the excellent quality of implementation.

Cite from the Disruptor documentation regarding queuing:
3.5 The Problems of Queues
[...]  If an in-memory queue is allowed to be unbounded then for many classes of problem it can grow unchecked until it reaches the point of catastrophic failure by exhausting memory. This happens when producers outpace the consumers. Unbounded queues can be useful in systems where the producers are guaranteed not to outpace the consumers and memory is a precious resource, but there is always a risk if this assumption doesn’t hold and queue grows without limit. [...]
When in use, queues are typically always close to full or close to empty due to the differences in pace between consumers and producers. They very rarely operate in a balanced middle ground where the rate of production and consumption is evenly matched. [...]
Couldn't have said it better myself =).



Conclusion


Although the Disruptor worked best for this example, I think looking for "the concurrency model to go for" is wrong. If we look at the real world, we see all 4 patterns used dependent on use case.
So a broad concurrency library ideally would integrate the assembly-line pattern (~Disruptor), queued messaging (~Actors) and unqueued communication (~CSP).



Benchmark source


AKKA




Multi Threading




Abstraktor




Disruptor naive




Disruptor optimized


203 comments:

  1. Rudiger, is hard to arrive to a conclusion. There are too many variables and misconceptions. But are we looking at the same thing. CSP and actors are both high level abstractions. And disruptor seems to me a very sophisticated queue implementation. Couldn't CSP and/or Actors be implemented with ring queues and memory barriers under the hood? Clearly platforms that rely too much on garbage collection may be at odds with the disruptor approach. But we could eventually see CSP and/or Actors catch up.

    ReplyDelete
  2. I don't think there is a general "best" model, it depends on use case.

    CSP has inherent problems with blocking, if one wants to avoid that, one has to accept indeterministic order of processing, which is a no go for many applications.
    Actors do not work well with any kind of bounded queue. As soon you limit queue size there is a deadlock scenario if your actor graph contains cycles.

    However the disruptor is more than a "queue" implementation. While actors need a queue for each processing unit following "move data not code", disruptor does this vice-versa "move code, not data". The analogy of a "mailbox" system and an "assembly line" pretty good illustrates the difference of the approaches. There are queues involved, but still the approach is quite diferrent.

    ReplyDelete
  3. And what about software transactional memory?

    ReplyDelete
    Replies
    1. There are no serious pure-software attempts for benchmarking available. However there are attempts to support this in hardware (Intel). I am not deep enough into the details to judge/guess viability of this approach. I'd assume it performes well for low contention but not so good in high contention cases. Additionally its not clear how and when the JVM will make this hardware feature avaiable.
      At time of writing I wasn't even aware of STM :-)

      Delete
  4. Useful read for me. Apart from throughput comparison I am also interested in know which concurrency model did you find easy to use and which one gives best safety guarantees. I am yet to dig to into concurrency models to reason about correctness and safety. Could you throw some light based on your experience?

    ReplyDelete
    Replies
    1. I think there is no "fits all" model.

      I found the Disruptor quite usable to setup high speed event/request processing engines, however it requires a different style to represent business logic and proper planning.
      As soon you hit some blocking API you'll need other ways to deal with concurrency.

      Actors: From my experience its manageable if you use it very course grained, as soon you setup complex actor systems with cyclic messaging/dependencies accross many actors, you'll run into serious trouble once real load is put on (queue overflows or deadlocks with bounded actor queues). Its by far not the "silver bullet" some folks try to make it.

      Try to think+design "reactive" e.g. a server component should be seen as a component processing a stream of incoming events single threaded (use NIO). If that is too slow, try to parallize sub-tasks, not the complete application flow. If that's still too slow, try partitioning. Avoid synchonous stuff (e.g. synchronous WebServices or remote calls, synchronous data base queries ). There are tricks to wrap blocking operations in order to avoid multithreading. Get used to queuing in order pass data from one processing unit to another.

      Unfortunately many of the old API's (e.g. servlets, jdbc) still are hard to use for a reactive style of programming and encourage/require creation of many threads without gaining anything but complexity.

      Delete
  5. This comment has been removed by the author.

    ReplyDelete
  6. Nice article, I think you have to consider reactive-streams as it is giving "asynchronous stream processing with non-blocking back pressure" which is meaning that it is reactive but consumer "subscriber" can control the pressure of the reactive calling, implementing dynamic push-pull way is giving benefits of reactive plus iterating, one of its implementations reactor using Disruptor implicitly.

    ReplyDelete
    Replies
    1. I implemented remoted reactive streams here: https://github.com/RuedigerMoeller/kontraktor/tree/trunk/modules/reactive-streams

      Though it might look "new" to many its in essence a simplified version of credit based flow control known from networking.
      A shortfall of current spec is the non-adaptive nature of "request(n)" pull sizes [tcp is better here] which is not an issue in-memory, but an issue at network grade latency. As the "request(n)" message has some travel time (depending on physical network conditions) the publisher might run dry. Its not too bad if libraries allow for manual tweaking, but not all allow for adjustment of "pull" size which leads to some whacky throughput variation when streaming over network.

      Delete
    2. Also note that Reactive Streams require queuing so its way slower than pure Disruptor processing. However its much easier to built up large scale, adaptive complex event processing pipelines using flow controlled Reactive Streams

      Delete
  7. Great article with clarity, Rüdiger! Just one question: how would you compare Asynchronous Programming to those paradigms? Thanks!

    ReplyDelete
  8. Hi, very clear. What about the "design" fact that in CSP one entity sends a message to a channel and another entity reads it from a channel, while in Actors actors sends directly messages one to another, so i can pass around the channel, it looks like a different abstractions which can be useful or not depending on use case, but in general having a channel I send events to sounds more reasnoable to me than objects talking directly (in many cases they should not know one another but when you read actor code you see that they are aware one of another a bad design by the fact of ease of misuse).

    ReplyDelete
  9. Devops is not a Tool.Devops Is a Practice, Methodology, Culture or process used in an Organization or Company for fast collaboration, integration and communication between Development and Operational Teams. In order to increase, automate the speed of productivity and delivery with reliability.

    python training in bangalore
    aws training in bangalore
    artificial intelligence training in bangalore
    data science training in bangalore
    machine learning training in bangalore
    hadoop training in bangalore
    devops training in bangalore

    ReplyDelete
  10. Gaining Python certifications will validate your skills and advance your career.
    python certification

    ReplyDelete
  11. Good job in presenting the correct content with the clear explanation. The content looks real with valid information. Good Work

    DevOps is currently a popular model currently organizations all over the world moving towards to it. Your post gave a clear idea about knowing the DevOps model and its importance.

    Good to learn about DevOps at this time.


    devops training in chennai | devops training in chennai with placement | devops training in chennai omr | devops training in velachery | devops training in chennai tambaram | devops institutes in chennai | devops certification in chennai

    ReplyDelete
  12. Devops is not a Tool.Devops Is a Practice, Methodology, Culture or process used in an Organization or Company for fast collaboration, integration and communication between Development and Operational Teams. In order to increase, automate the speed of productivity and delivery with reliability.



    digital marketing courses in Bangalore With placement

    digital marketing training in Bangalore

    seo training in Bangalore

    ReplyDelete
  13. Very Nice Information! Brainy helps children's to develop their creativity and social skills.
    At Brainy, There are many child brain development activities.
    Also you can apply for educational franchise opportunity India.

    ReplyDelete
  14. Maniry anao foana ny fifaliana sy ny fahasambarana. Manantena ianao fa manana lahatsoratra tsara kokoa ianao.

    Phối chó bull pháp

    Phối giống chó Corgi

    Phối chó Pug

    Phối giống chó alaska

    ReplyDelete
  15. This is really a big and great source of information. We can all contribute and benefit from reading as well as gaining knowledge from this content just amazing
    experience Thanks for sharing such a nice information.

    DedicatedHosting4u.com

    ReplyDelete
  16. Thanks for the informative article. This is one of the best resources I have found in quite some time. Nicely written and great info. I really cannot thank you enough for sharing.
    Sql server dba online training

    ReplyDelete
  17. It is a very popular application which provides you to download videos, songs, mashups, remixes, trailers in your mobile , and off course in any quality. It contains many videos and of many types .
    like funny videos, motivational videos, songs lyrics, trailer and many more.
    Want to watch movie in slow connection,
    Vidmate app,
    Vidmate app to watch videos,
    vidmate,
    Vidmate Features,
    Vidmate Download,
    Multi sSource Downloader,
    cm security antivirus,
    leaked Movie Online,
    Why do people prefer Vidmate App.

    ReplyDelete
  18. This is a nice Site to watch out for and we provided information on
    vidmate make sure you can check it out and keep on visiting our Site.

    ReplyDelete
  19. Download and install Vidmate App which is the best HD video downloader software available for Android. Get free latest HD movies, songs, and your favorite TV shows.

    ReplyDelete
  20. This comment has been removed by the author.

    ReplyDelete
  21. Download latest audio and video file fromvidmate

    ReplyDelete
  22. Download latest audio and video file fromvidmate

    ReplyDelete
  23. Download latest audio and video file fromvidmate

    ReplyDelete
  24. Thanks for one marvelous posting! I enjoyed reading it; you are a great author. I will make sure to bookmark your blog and may come back someday. I want to encourage that you continue your great posts...
    Cloud Computing Online Training

    ReplyDelete
  25. I really like your blog. You make it interesting to read and entertaining at the same time. I cant wait to read more from you.
    salesforce online Certification

    ReplyDelete

  26. Nuvigil smart drug is the trade name for Armodafinil smart drug and is the enantiopure compound of Modafinil. There are a number of health benefits that the use of Nuvigil smart drug has to offer to its users. So, you can buy Nuvigil online. The use of the smart drug is safe and has no health issues attached to its usage. In the year 2006, the FDA approved the use of generic Nuvigil smart drug to be used to treat a number of mental issues. Buy Nuvigil online as the use of Nuvigil smart drug has long effect when compared to other smart drugs. This is mainly because Armodafinil smart drug has a longer half-life than the other smart drugs. Therefore more and more people are depending on Nuvigil smart drug to carry on their daily routine smoothly.
    Buy Nuvigil online

    ReplyDelete
  27. Really i found this article more informative, thanks for sharing this article! Also Check here

    Download and install Vidmate App which is the best HD video downloader software available for Android. Get free latest HD movies, songs, and your favorite TV shows

    Vidmate App Download

    Vidmate apk for Android devices

    Vidmate App

    download Vidmate for Windows PC

    download Vidmate for Windows PC Free

    Vidmate Download for Windows 10

    Download Vidmate for iOS

    Download Vidmate for Blackberry

    Vidmate For IOS and Blackberry OS

    ReplyDelete
  28. After looking at a number of the blog articles on your web site, I seriously appreciate your technique of blogging. I bookmarked it to my bookmark website list and will be checking back soon. Take a look at my website as well and let me know your opinion. motorola display repair bangaloreThis web site certainly has all the information I wanted concerning this subject and didn’t know who to ask. lg service center Bangalore There is definately a lot to find out about this topic. I like all of the points you made. vivo charging port replacement

    ReplyDelete
  29. I used to be able to find good information from your content. huawei display repair bangalore Great post! We will be linking to this particularly great article on our site. Keep up the good writing. asus display replacement This is a great tip particularly to those new to the blogosphere. Short but very accurate information… Thank you for sharing this one. A must read article! onsite mobile repair bangalore

    ReplyDelete


  30. Whatever we gathered information from the blogs, we should implement that in practically then only we can understand that exact thing clearly, but it’s no need to do it, because you have explained the concepts very well. It was crystal clear.i also want to inform you the best salesforce developer . thankyou . keep sharing..

    ReplyDelete
  31. Poker online situs terbaik yang kini dapat dimainkan seperti Bandar Poker yang menyediakan beberapa situs lainnya seperti http://62.171.128.49/hondaqq/ , kemudian http://62.171.128.49/gesitqq/, http://62.171.128.49/gelangqq/, dan http://62.171.128.49/seniqq. yang paling akhir yaitu http://62.171.128.49/pokerwalet/. Jangan lupa mendaftar di panenqq silakan dicoba ya boss

    ReplyDelete
  32. I admire the valuable information you offer in your articles. I learn a lot of new things here
    Node JS Online training
    Node JS training in Hyderabad

    ReplyDelete
  33. This post is really nice and informative. The explanation given is really comprehensive and informative.I want to inform you about the salesforce business analyst training and video tutorials . thankyou . keep sharing..

    ReplyDelete
  34. It is amazing and wonderful to visit your site.Thanks for sharing this information,this is useful to me...
    http://chennaitraining.in/informatica-data-quality-training-in-chennai/
    http://chennaitraining.in/informatica-iics-training-in-chennai/
    http://chennaitraining.in/talend-training-in-chennai/
    http://chennaitraining.in/power-bi-training-in-chennai/
    http://chennaitraining.in/sap-bo-training-in-chennai/
    http://chennaitraining.in/qlikview-training-in-chennai/

    ReplyDelete
  35. Thank you for taking the time to provide us with your valuable information. We strive to provide our candidates with excellent care
    http://trainingsinvelachery.in/sap-hr-training-in-velachery/
    http://trainingsinvelachery.in/sap-mm-training-in-velachery/
    http://trainingsinvelachery.in/sap-sd-training-in-velachery/
    http://trainingsinvelachery.in/sap-fico-training-in-velachery/
    http://trainingsinvelachery.in/sap-abap-training-in-velachery/
    http://trainingsinvelachery.in/sap-hana-training-in-velachery/

    ReplyDelete
  36. Thanks for Sharing This Article.It is very so much valuable content. I hope these Commenting lists will help to my website
    microservices online training
    best microservices online training
    top microservices online training

    ReplyDelete
  37. Hello Admin!

    Thanks for the post. It was very interesting and meaningful. I really appreciate it! Keep updating stuffs like this. If you are looking for the Advertising Agency in Chennai | Printing in Chennai , Visit Inoventic Creative Agency Today..

    ReplyDelete
  38. Thanks for Sharing This Article.It is very so much valuable content. I hope these Commenting lists will help to my website
    welcome to akilmanati
    akilmanati

    ReplyDelete
  39. It's really a nice and useful piece of information about Cloud Computing. I'm satisfied that you shared this helpful information with us.Please keep us informed like this. Thank you for sharing.

    Java training in chennai | Java training in annanagar | Java training in omr | Java training in porur | Java training in tambaram | Java training in velachery

    ReplyDelete
  40. A backlink is a link created when one website links to another. Backlinks are important to SEO & impact for higher ranking. In my 7+ years seo Career i see, without backlinks a website doesn't rank higher on google SERP.

    Get Your 300+ High Quality DoFollow Backlinks Here!

    Order Now with Full Confidence & 100% satisfaction.

    ReplyDelete
  41. Thanks for sharing the great information, I hope really enjoyed while reading the article.
    Java Online Training
    Python Online Training
    PHP Online Training

    ReplyDelete
  42. Forex Signals, MT4 and MT5 Indicators, Strategies, Expert Advisors, Forex News, Technical Analysis and Trade Updates in the FOREX IN WORLD

    Forex Signals Forex Strategies Forex Indicators Forex News Forex World

    ReplyDelete
  43. Thanks for Sharing This Article.It is very so much valuable content. I hope these Commenting lists will help to my website
    workday studio online training
    best workday studio online training
    top workday studio online training

    ReplyDelete
  44. AGEN BOLA TERPERCAYA
    KAMUBET – SITUS JUDI BOLA, TOGEL ONLINE, DAN AGEN CASINO

    ======SPORTBOOK======
    - SBOBET | AFB88 | IBCBET -
    Minimal Bet Parlay : Rp 3.000
    Minimal Single Bet : Rp 10.000

    ======DISKON TOGEL======
    2D – 29.75% X 70
    3D – 59.50% X 400
    4D – 66.00% X 3000

    ======PROMO=====
    BONUS WELCOME DEPOSIT 50% SLOT GAMES
    BONUS NEXT DEPOSIT 10%
    BONUS ROLLINGAN CASINO 1%
    BONUS CASHBACK SLOT GAMES UP TO 5%
    BONUS CASHBACK SPORTBOOK 10%

    =========================
    Minimal Deposit : Rp 10.000
    Minimal Withdraw : Rp 50.000
    Minimal Deposit Pulsa : Rp. 15.000
    Proses Deposit dan Withdraw hanya 2 Menit

    ======HUBUNGI KAMI======
    Link : 178.128.120.230
    Whatsapp - +855882285275
    Instagram : @kamubet77
    Facebook : @kamubet888
    Twitter : @kamubet
    Line – Kamubet
    Wechat – Kamubet

    ReplyDelete
  45. Slot Genting - Agen Slot Online Uang Asli

    ======PROMO=====
    BONUS WELCOME DEPOSIT 50%
    BONUS NEXT DEPOSIT 20%
    BONUS CASHBACK 5%

    =========================
    Minimal Deposit : Rp 10.000,-
    Minimal Withdraw : Rp 50.000,-
    Minimal Deposit Pulsa : Rp 15.000,-
    Proses Deposit dan Withdraw hanya 2 Menit

    ======HUBUNGI KAMI======
    Link Resmi – slotgenting
    Whatsapp +6285740170865
    LIVECHAT ONLINE 24/7
    FB : SlotGenting888

    ReplyDelete
  46. Thanks for Sharing This Article.It is very so much valuable content. I hope these Commenting lists will help to my website
    akilmanati

    ReplyDelete
  47. The great thing about this post is quality information. I always like to read amazingly useful and quality content. Your article is amazing, thank you for sharing this article.
    tata cliq offers
    mi offers
    swiggy offers
    zomato coupons

    ReplyDelete
  48. Thanks for Sharing This Article.It is very so much valuable content. I hope these Commenting lists will help to my website
    akilmanati

    ReplyDelete
  49. Thanks for Sharing This Article.It is very so much valuable content. I hope these Commenting lists will help to my website
    blockchain online training
    best blockchain online training
    top blockchain online training

    ReplyDelete
  50. Shield Security Solutions Provides Ontario Security Training, Security Guard License or Security License in Ontario. Get Started Today

    ReplyDelete
  51. yahoo mail forgot passwordYahoo mail forgot password Are you one of them who forgot their string of secret characters, don't panic you are counted in that of users who forgot password. and solve for this problem.

    ReplyDelete
  52. Thank you for sharing this informative article with us. I got lots of information from this website. Keep sharing more article like this.

    ReplyDelete
  53. There are many theology & religion coursework writing services and Religious Research Writing Services to choose from for those stuck with their religion assignment writing services and theology essay writing help services.

    ReplyDelete
  54. very interesting , good job and thanks for sharing such a good blog. Seo Services Delhi

    ReplyDelete
  55. very interesting, good job and thanks for sharing such a good blog. Youtube Mp3 Converter

    ReplyDelete
  56. I just loved your article on the beginners guide to starting a blog.If somebody take this blog article seriously
    in their life, he/she can earn his living by doing blogging.Thank you for this article.
    best java online training

    ReplyDelete
  57. I just loved your article on the beginners guide to starting a blog.If somebody take this blog article seriously
    in their life, he/she can earn his living by doing blogging.Thank you for this article.
    best java online training

    ReplyDelete
  58. I just loved your article on the beginners guide to starting a blog.If somebody take this blog article seriously
    in their life, he/she can earn his living by doing blogging.Thank you for this article.
    best java online training

    ReplyDelete
  59. You have a big talent! Don't waste it, I advise you to create youtube channel too and post video. On this site https://soclikes.com/ you can get cheap youtube subscribers, buy at least your first subscribers

    ReplyDelete
  60. It's going on in the digital era and have no time to sell a product via face to face. So we are using an online marketplace to sell our products. Quikads is such a platform where you can sell your used phone & other products easily. It reduces our time and makes our life so easy and comfortable.

    ReplyDelete
  61. jungle ka paryayvachi – जंगल शब्द का पर्यायवाची है (The word jungle is synonymous -)–

    ReplyDelete
  62. If you are a youtuber, then know How To Get Sponsorship For YouTube Channel for free!

    ReplyDelete
  63. Thanks for the interesting content. I like your post and your blog is amazing.
    If you are interested in Video Downloader apps you can check my blog site. It is new and really informative.

    VidMate Download 2018

    ReplyDelete
  64. Kim Ravida is a lifestyle and business coach who helps women in business take powerful money actions and make solid, productiveIamLinkfeeder IamLinkfeeder IamLinkfeeder IamLinkfeeder IamLinkfeeder IamLinkfeeder IamLinkfeeder IamLinkfeeder IamLinkfeeder IamLinkfeeder

    ReplyDelete
  65. Besides my C course, I have a job and family, both of which compete to get my time. I couldn't find sufficient time for the challenging C assignments, and these people came in and saved my skin. I must commend them for the genius Programming Assignment Help. Their C Homework Help tutors did the best job and got me shining grades.

    ReplyDelete
  66. I am looking for a Statistics Assignment Help expert for Statistics Homework Help. I have struggled enough with statistics and therefore I just can't do it anymore on my own. . I have come across your post and I think you are the right person to provide me with SPSS homework help. Let me know how much you charge per assignment so that I can hire you today

    ReplyDelete
  67. Matlab Assignment Help helped me to complete my seventh Matlab assignment, which was also the best-performed! It scored 92/100, which I've never scored before on any other assignment/exam in my lifetime. Otherwise, their service was as quick as usual. The delivery was also on time. I'm now requesting to use this same programmer multiple times. He seems the best in Image Processing tasks. Meanwhile, I'll ask for more Matlab Homework Help soon.

    ReplyDelete
  68. I have just come across your post and I believe this is exactly what I am looking for. I want an economics assignment help from a tutor who can guarantee me a top grade. Do you charge per page or does it depend on the
    bulk of the economics homework help being completed? More to that if the work is not good enough do you offer free corrections.

    ReplyDelete
  69. Hi, other than economics assignment help are there other subjects that you cover? I am having several assignments one needs an economics homework help expert and the other one needs a financial expert. If you can guarantee quality work on both then I can hire you to complete them. All I am sure of is that I can hire you for the economics one but the finance one I am not sure.

    ReplyDelete
  70. Me and my classmates took too long to understand Matlab Assignment Help pricing criteria. we're always grateful for unique solutions on their Matlab assignments. Matlab Homework Help experts have the right experience and qualifications to work on any programming student's homework. They help us in our project.

    ReplyDelete
  71. The ardent Programming Homework Help tutor that nailed down my project was very passionate. He answered my Python questions with long, self-explanatory solutions that make it easy for any average student to revise. Moreover, he didn't hesitate to answer other questions, too, even though they weren't part of the exam. If all Python Homework Help experts can be like this then they can trend as the best Programming school ever online.

    ReplyDelete
  72. Hey STATA homework help expert, I need to know if you can conduct the Kappa measurement of agreement. This is what is in my assignment. I can only hire someone for statistics assignment help if they are aware of the kappa measurement of agreement. If you can do it, then reply to me with a few lines of what the kappa measure of agreement is. Let me know also how much you charge for statistics homework help in SAS.

    ReplyDelete
  73. Our Data Science course in Hyderabad will also help in seeking the highest paid job as we assist individuals for career advancement and transformation. We carefully curate the course curriculum to ensure that the individual is taught the advanced concepts of data science. This helps them in solving any challenge that occurs. Along with that, we also make students work on real case studies and use-cases derived.

    data science course in hyderabad
    data science Training in hyderabad

    ReplyDelete
  74. Use Assignment Help services to finish your work effectively. If you are facing issues in writing your academic papers, you must use an online source of expert help. It is quite obvious to pay someone to do my assignment seem difficult to you when you don’t have complete information about the service provider. So, it is good to dig out the full details of your selected service provider and pay the price as soon as possible.

    ReplyDelete
  75. Hey there, I need an Statistics Homework Help expert to help me understand the topic of piecewise regression. In our lectures, the concept seemed very hard, and I could not understand it completely. I need someone who can explain to me in a simpler way that I can understand the topic. he/she should explain to me which is the best model, the best data before the model and how to fit the model using SPSS. If you can deliver quality work then you would be my official Statistics Assignment Help partner.

    ReplyDelete
  76. I don’t have time to look for another expert and therefore I am going to hire you with the hope that I will get qualityeconomics assignment help. .Being aneconomics homework help professor I expect that your solutions are first class. All I want to tell you is that if the solutions are not up to the mark I am going to cancel the project.

    ReplyDelete
  77. That is a huge number of students. Are they from the same country or different countries? I also want your math assignment help. I want to perform in my assignments and since this is what you have been doing for years, I believe you are the right person for me. Let me know how much you charge for your math homework help services.

    ReplyDelete
  78. Hello. Please check the task I have just sent and reply as soon as possible. I want an adjustment assignment done within a period of one week. I have worked with an Accounting Homework Help tutor from your team and therefore I know it’s possible to complete it within that period. Let me know the cost so that I can settle it now as your Accounting Assignment Help experts work on it.

    ReplyDelete
  79. I got very good information in this post, thank you very much from my side, keep giving this kind of information.
    1.Hindi studio

    ReplyDelete
  80. Looking for amazing and latest technical updates?
    Do visit us at
    https://geekermag.com/

    ReplyDelete
  81. Thanks for crafting this beautiful information. I’m thankful and humbled to you. This is the best blog I have ever read. I’m going to make sure that I will share. Rubix Market Research

    ReplyDelete
  82. Bookkeeping services In Australia
    Leave your manual bookkeeping services stress to our experts at OrangeIQ. Outsource bookkeeping services in Australia with OrangeIQ

    ReplyDelete
  83. Great content & Thanks for sharing with uttar pradesh's YouTuber ibrahim 420

    ReplyDelete
  84. Thats great post !! I like ur every post they always give me some new knowledge.

    VidMate | VidMate for PC |
    VidMate 2014

    ReplyDelete
  85. This comment has been removed by the author.

    ReplyDelete
  86. Thanks for the interesting content. I like your post and your blog is amazing.
    If you are interested in Video Downloader apps you can check my blog site. It is new and really informative.

    utorrent pro mod apk free download

    ReplyDelete
  87. Thanks for the interesting content. I like your post and your blog is amazing.
    If you are interested in Video Downloader apps you can check my blog site. It is new and really informative.

    shareit

    ReplyDelete
  88. Windows Live Hotmail is the second popular email service available in the world, after Yahoo! Yes, it is even more popular than Gmail, given Microsoft is the developer of it. Hotmail comes in two different versions, free and paid. Here is the complete gudie on hotmail.com.signin process.

    ReplyDelete
  89. How to make yahoo my homepage on Firefox?

    If you have no idea about how to make yahoo my homepage on Firefox, go through the steps mentioned here . First of all, launch the Firefox browser and then click on the Menu icon . Choose the Preferences icon under it. Now, under the General page, click on Show my home page option. Next, under the Home Page field, you need to enter www.Yahoo.com. Now, Yahoo will be made as your default homepage for Firefox browser.




    ReplyDelete
  90. Situs Nonton movie, film dan tv series terbaru dengan subtitle indonesia diupdate setiap hari, dari situs terpopuler nonton disini link
    di bawah ini
    layarkaca21
    bioskopkeren
    daymovie
    terbit21
    boomthis

    Situs judi online terpercaya
    http://199.188.201.133
    http://162.213.251.13
    bonekaqq
    indoqq99
    sahabatqq

    ReplyDelete
  91. This is a very nice one and gives in-depth information. I am really happy with the quality and presentation of the article. I’d really like to appreciate the efforts you get with writing this post. Thanks for sharing.
    Java Course in pune

    ReplyDelete
  92. toptan iç giyim tercih etmenizin sebebi kaliteyi ucuza satın alabilmektir. Ürünler yine orjinaldir ve size sorun yaşatmaz. Yine de bilinen tekstil markalarını tercih etmelisiniz.

    Digitürk başvuru güncel adresine hoşgeldiniz. Hemen başvuru yaparsanız anında kurulum yapmaktayız.

    tutku iç giyim Türkiye'nin önde gelen iç giyim markalarından birisi olmasının yanı sıra en çok satan markalardan birisidir. Ürünleri hem çok kalitelidir hem de pamuk kullanımı daha fazladır.

    nbb sütyen hem kaliteli hem de uygun fiyatlı sütyenler üretmektedir. Sütyene ek olarak sütyen takımı ve jartiyer gibi ürünleri de mevcuttur. Özellikle Avrupa ve Orta Doğu'da çokça tercih edilmektedir.

    yeni inci sütyen kaliteyi ucuz olarak sizlere ulaştırmaktadır. Çok çeşitli sütyen varyantları mevcuttur. iç giyime damga vuran markalardan biridir ve genellikle Avrupa'da ismi sıklıkla duyulur.

    iç giyim ürünlerine her zaman dikkat etmemiz gerekmektedir. Üretimde kullanılan malzemelerin kullanım oranları, kumaşın esnekliği, çekmezlik testi gibi birçok unsuru aynı anda değerlendirerek seçim yapmalıyız.

    iç giyim bayanların erkeklere göre daha dikkatli oldukları bir alandır. Erkeklere göre daha özenli ve daha seçici davranırlar. Biliyorlar ki iç giyimde kullandıkları şeyler kafalarındaki ve ruhlarındaki özellikleri dışa vururlar.

    ReplyDelete
  93. java has really become the new c, as it has taken over in many domains.
    just like gmail register

    ReplyDelete
  94. Airpods have become a part of our lives. I don’t see anyone owning an iPhone that does not use AirPods. It is easy to use, carry anywhere, and even charge. Usually, the battery life of the first- and second-generation AirPods on a single charge lasts for around 5 hours while the user is listening to music and is 3 hours while having a phone conversation. Here is the complete guest for check airpods battery.

    ReplyDelete
  95. This information is really awesome thanks for sharing most valuable information.
    workday training online
    workday software training

    ReplyDelete
  96. Thanks for this post it is really superb and amazing post it is very informative post. Fashion Bloggers In India

    ReplyDelete
  97. I recently came across your article and have been reading along. I want to express my admiration of your writing skill and ability to make readers read from the beginning to the end.
    Java Classes in Pune

    ReplyDelete
  98. Much of our Sofa Maintaining Services is amazingly unique. We make use of different variety of tools to help you sanitize any corner from your sofa. You use choose to dry memory foam or expressly formalized soap solution designed for sensitization assuring optimum gains. The drying out process can take maximum 3 hours additionally, the couch decide to use within too busy. We you should not use any specific chemicals to help you spoil ones own fabric and risk yourself. We Have got unmatched high-quality Upholstery Maintaining Services on Dubai. cleaning services in dubai

    ReplyDelete
  99. Data Science Course with R, Python & 15+ Projects. Ranked world's #1 Online Bootcamp. Get Noticed by the Top Hiring Companies with Bygrad Job Guaranteed Program.
    Online Data Science Program

    ReplyDelete
  100. According to the official statement, IPL 2nd phase is going be launched from 19th September. There are 31 remaining matches pending and they will be continued till 10th October, the day final match will be played. Check out Vivo IPL 2021 UAE Schedule, IPL UAE Match Timings in india, Points Table, Team positions & Player stats in this article.

    ReplyDelete
  101. Are you in a confusion in choosing the best option between renting a laptop & owning a laptop ?Before talking about the renting a laptop, once imagine yourself in a situation, I am about to mention.Amateur or an expert you will still google and find out what laptops you can get depending on the budget, need, offers.

    ReplyDelete
  102. These comparison is amazing and good for the knowledge purpose.
    Movierulz4

    ReplyDelete
  103. K.G.F: Chapter 2 : Directed by Prashanth Neel. With Yash, Sanjay Dutt, Raveena Tandon, Prakash Raj. The blood-soaked land of Kolar Gold Fields

    ReplyDelete
  104. Thanks for Sharing This Article.It is very so much valuable content. I hope these Commenting lists will help to my website mulesoft online training
    best mulesoft online training
    top mulesoft online training

    ReplyDelete
  105. Do you need an urgent loan of any kind? Loans to liquidate debts or need to loan to improve your business have you been rejected by any other banks and financial institutions? Do you need a loan or a mortgage? This is the place to look, we are here to solve all your financial problems. We borrow money for the public. Need financial help with a bad credit in need of money. To pay for a commercial investment at a reasonable rate of 3%, let me use this method to inform you that we are providing reliable and helpful assistance and we will be ready to lend you. Contact us today by email: daveloganloanfirm@gmail.com Call/Text: +1(501)800-0690 And whatsapp: +1 (501) 214‑1395

    NEED A LOAN?
    Ask Me.

    ReplyDelete
  106. Hello Sir I saw your blog, It was very nice blog, and your blog content is awesome, i read it and i am impressed of your blog, i read your more blogs, thanks for share this summary.
    How to change yahoo mail password

    ReplyDelete
  107. I have to convey my respect for your kindness for all those that require guidance on this one field. Your special commitment to passing the solution up and down has been incredibly functional and has continually empowered most people just like me to achieve their dreams. Your amazing insightful information entails much to me and especially to my peers.
    hướng dẫn đi máy bay từ mỹ về việt nam

    Máy bay từ Hàn Quốc về Việt Nam

    Ve may bay Bamboo tu Nhat Ban ve Viet Nam

    vé máy bay từ singapore về hà nội vietjet

    Bảng giá vé máy bay Vietjet Air tu Dai Loan ve Viet Nam

    thông tin chuyến bay từ canada về việt nam

    ReplyDelete
  108. vst4crack comes with a simple and friendly interface, allowing you to optimize your system registry with just a few mouse clicks.








    ReplyDelete
  109. Download usd-rmb Mt4: A Complete Solution For Your Forex Trading.

    ReplyDelete
  110. Thanks for sharing valuable article having good information and also gain worthful knowledge. We are also providing the best services click on below links to visit our website.
    Oracle Fusion HCM Training
    Workday Training
    Okta Training
    Palo Alto Training
    Adobe Analytics Training

    ReplyDelete
  111. https://koladblog.com.ng/2021/10/08/top-6-really-free-vpn-for-android/

    https://koladblog.com.ng/2021/10/16/how-to-apply-for-opay-pos-in-nigeria/

    opay pos

    ReplyDelete
  112. Thank you for nice information, Visit Us:
    Here

    ReplyDelete
  113. I’ve been following your web site for some time now and finally, I decided to write this post in order to thank you for providing us such a fine content!!! Feel free to visit my website; 바카라사이트

    ReplyDelete