WCF Security token in the message could not be validated when using Custom authentication

The security token error is usually generated from a SecurityTokenValidation exception during the Username and password validation process.   This error is usually displayed as follows:

[MessageSecurityException: An unsecured or incorrectly secured fault was received from the other party. This fault may have been sent in response to an improperly secured request. See the inner FaultException for the fault code and detail.]

The inner exception displays this error:

At least one security token in the message could not be validated.

This error does not tell us much about the problem, but we know there is a security problem. A way to troubleshoot this is to first enable logging on the web service. We enable the logging of the messages at the transport level by adding the following diagnostics settings in your web.config file inside the system.serviceModel node.
<messageLogging maxMessagesToLog="25000" logEntireMessage="true" logMessagesAtServiceLevel="false"  logMalformedMessages="true" logMessagesAtTransportLevel="true">
We now need to create a file that we can open with Microsoft Trace viewer by adding this setting under the system.diagnostic node: (notice the name and location of the log file)
<source name="System.ServiceModel" switchValue="Warning, ActivityTracing" propagateActivity="true" >
<listeners> <add name="xml" /></listeners>
<source name="System.ServiceModel.MessageLogging" switchValue="Warning">
   <listeners><add name="xml" /></listeners>
   <add name="xml" type="System.Diagnostics.XmlWriterTraceListener" initializeData="C:\Temp\myService.svclog" />
You can now deploy the web.config changes in your test environment and send a couple of request. If you do this on production, make sure to remove the settings after recreating the problem. Once you have recreated the problem, look for the log file (see initializeData attribute) and open it with trace viewer. We are looking for items highlighted in red under the Activity tab (left pane). Select one of those messages and on the right-top pane look for a message that reads "Throwing an exception". Select that message and inspect the details under the XML tab(see image below). There, we can find a detail stack dump of what is really causing the problem. You can look carefully for the class name in your project, and you would probably be able to identify exactly what may have gone wrong.
I hope this helps.