scodec
Scala combinator library for working with binary data
scodec - Home
What is the best way to read and write C-styled byte stuctures in Scala, like this:
struct account {
int id;
char[10] data1;
char[10] data2;
float dataFloat;
};
There's unpack function in Python, to interpret strings as packed binary data. But I can't find any analogue in Scala.
What is standart way for such a mapping in Scala? Read bytes one by one is very unsufficiant. The protocol I need to parse comes back fromk 1980s and contains different fields (short, int, float) so read it byte-by-byte will be very unsufficient.
Source: (StackOverflow)
I must create a codec for a message that has the following specification
The message length is indicated by a byte of which the least significant bit is an extension flag that, when set indicates that the following (optional) byte must be used as the most significant byte. (Hope it make sense) It can be depicted as follows:
+----------------------------------------------------------------------------------+
| LENGTH |
| |
+----------------------------------+-----+-+---------------------------------------+
| | | | |
| Len1 (LSB) | Ext | | Len2 (MSB) - Optional |
+----+----+----+----+----+----+----+-----+ +----+----+----+----+----+----+----+----+
| | | | | | | | | | | | | | | | | |
| | | | | | | | + | | | | | | | | | |
+----+----+----+----+----+----+----+--|--+ +----+----+----+----+----+----+----+----+
|
|
v
Boolean: if true then Len2 is used
else only Len1
The length of the data that will follow is determined by this field(s). I would like to use the codec along with predefined codecs and combinators.
I guess it will involve using flatZip but I am not clear on how to incorporate flatZip into an HList combinator.
Any pointers to examples or documentation will be much appreciated.
Source: (StackOverflow)
I am using scodec: https://github.com/scodec/scodec to decode/encode a binary protocol.
I am struggling with a part of the spec where a "length" field is split into two parts by a "moreflag". The moreflag indicates if the length field needs more space.
Example:
Byte 1: identifier bits 8-7, moreFlag bit 6, length bits 5-0 // first length field filled with 0's if moreFlag is false
Byte 2: moreFlag bit 8, length bits 7-0
Byte 3: otherJunk bits 8-0
My problem is I want to encode/decode both of these length fields into a single case class field:
case class Header(totalLength: Int, otherJunk: Int)
I have tried a few different things, but nothing has worked out so far:
implicit val headerCodec: Codec[Header] = (
("identifier" | uint2) :~>:
("moreFlag" | bool).compact >>:~ { meta =>
if (meta.last) {
// more flag enabled, combine lengths somehow
("first length part" | uint(5)) :: ("moreFlag2DontCare" | uint(1) :~>: ("second length part - how to combine?" | uint(7)) :: ("otherJunk" | uint8)
}
else {
("first length part always 0s" | constant(bin"00000")) :: ("moreFlag2DontCare" | uint(1) :~>: ("fullLength" | uint(7)) :: ("otherJunk" | uint8)
}
}
).as[Header]
Am I on the right track here? Thanks!
Source: (StackOverflow)
I am trying to define an Scodec coproduct codec for communicating with an EELink GPS.
Here is the code:
import scodec.Codec
import scodec.bits.ByteVector
import scodec.codecs._
trait Message
object Message {
implicit val discriminated: Discriminated[ Message, Int ] = Discriminated(uint8)
val codec: Codec[ Message ] = Codec.coproduct[ Message ].discriminatedByIndex(uint8)
}
case class GpsId(value: ByteVector)
object GpsId {
val codec = bytes(8).as[ GpsId ]
}
case class SerialNumber(value: Int)
object SerialNumber {
val codec = uint16.as[ SerialNumber ]
}
case class Header(protocolNumber: Int, length: Int, serial: SerialNumber)
object Header {
val codec = (uint8 :: uint16 :: SerialNumber.codec).as[ Header ]
}
case class Login(header: Header, id: GpsId, language: Int) extends Message
object Login {
val protocolNumber = 0x01
implicit val discriminator: Discriminator[ Message, Login, Int ] = Discriminator(protocolNumber)
implicit val codec: Codec[Login] = (Header.codec :: GpsId.codec :: uint8).as[ Login ]
}
I am getting the following:
Error:(14, 48) could not find implicit value for parameter auto: scodec.codecs.CoproductBuilderAuto[com.tecnoguru.ridespark.gps.eelink.messages.Message]
val codec: Codec[ Message ] = Codec.coproduct[ Message ].discriminatedByIndex(uint8)
^
I have looked at Scodec - Coproducts could not find implicit value for parameter auto: scodec.codecs.CoproductBuilderAuto but it did not help, from what I see I am defining the codec and the discriminator correctly.
I am running on Scala 2.11.5 with scodec-core 1.7.0 and scodec-bits 1.0.5
Source: (StackOverflow)
I'm just starting out with typelevel's "scodec" library: https://github.com/scodec/scodec
I've found that I've been using the following function a lot:
/**
* When called on a `Codec[L]` for some `L <: HList`, returns a new codec that encodes/decodes
* `B :: L` but only returns `L`. HList equivalent of `~>`.
* @group hlist
*/
def :~>:[B](codec: Codec[B])(implicit ev: Unit =:= B): Codec[L] = codec.dropLeft(self)
This is useful if I have a case class where I don't want to use every value of the spec:
case class Example(value1: Int, value3)
implicit val exampleCodec: Codec[Example] = (
("value1" | uint8) ::
("value2" | uint8) :~>: // decode/encode, but dont pass this in when converting from hlist to case class
("value3" | uint8)
).as[Example]
This works well if the value I want to ignore isn't the last one in the hlist. Would anyone know how to change the codec, if instead I wanted my case class to be:
case class Example(value1: Int, value2: Int) // ignore value3
Any help is appreciated - thanks!
Source: (StackOverflow)
For "big" codecs, the Scala phase typer takes forever (we're talking minutes) when creating a codec directly from HLists and applying .dropUnits
( ignore(6) ::
uint(2) ::
uint(30) ::
int(4) ::
int(8) ::
uint(10) ::
bool(1) ::
int(28) ::
int(27) ::
uint(12) ::
uint(9) ::
uint(6) ::
int(2) ::
ignore(3) ::
bool(1) ::
uint(19)
).dropUnits.as[SomeBigCaseClass]
And it seems to be way faster to create a codec with ~, and then applying .hlist like such:
( ignore(6) ~
uint(2) ~
...
).hlist.dropUnits.as[SomeBigCaseClass]
But this doesn't seem work.
Could not prove that this.Out can be converted to/from reports.SomeBigCaseClass.
).hlist.dropUnits.as[SomeBigCaseClass]
^
The simplest solution I've found, which is good enough for me is omitting Unit values inline.
( (ignore(6) dropLeft
uint(2)) ::
...
).as[SomeBigCaseClass]
For codecs with many ignores, this feature would be highly welcome. What am I doing wrong? Am I totally missing the point of .hlist?
Source: (StackOverflow)
I have a class looking like this,
case class Foo ( bar: Int, foos: Vector[Foo] )
to define a Codec[Foo], I tried this,
def fc = shapeless.Lazy((int32 ~~ vector(fc)).widenOpt( (Foo.apply _).tupled, Foo.unapply _ ))
But this did not work, since scodec throws StackOverflowError. What is the right way of doing this ?
Source: (StackOverflow)
I'm trying to use scodec to decode a list of records where the records are identified by a separate list of record identifiers.
The binary contains a list of integers where the position of an integer in the list and its value represents the order and type of the record in a subsequent list of values.
For example, if 1 :: 2 :: 3 is the identifier list, then the value list would be { type 1 values } :: { type 2 values } :: { type 3 values }
The type list can contain any number and order of a fixed set of types.
I've been trying to figure out a way to build a shapeless Hlist, but I only know the type list at run-time so I don't think that is an option. Any suggestions?
Source: (StackOverflow)
When defining my messages with scodec, I would like to use nested case classes. For example:
case class Foo(x: Int, y: Int)
object Foo {
def baseCodec = uint16 :: uint16
def codec = baseCodec.as[Foo]
}
case class Bar(a: Int, foo: Foo, b: Int)
object Bar {
val baseCodec = uint8 :: Foo.baseCodec :: uint16
val codec = baseCodec.as[Bar]
}
However, when trying to compile this I get the following:
error: Could not prove that shapeless.::[Int,shapeless.::[shapeless.::[Int,shapeless.::[Int,shapeless.HNil]],shapeless.::[Int,shapeless.HNil]]] can be converted to/from Bar.
val codec = baseCodec.as[Bar]
^
Is there a way of doing this? (In my real code, sometimes the nested case class appears at the beginning of the containing class' parameter list, sometimes in the middle and sometimes at the end).
Source: (StackOverflow)
I'm trying to write a combinator for the scodec library that converts a Codec[K] in to a Codec[L] where K is an HList and L is the equivalent HList with all Unit elements removed.
Implementing decoding can be done by decoding a K and then filtering out all Unit elements. Filtering out Unit elements is directly supported by shapeless using filterNot, which makes this trivial to implement.
Implementing encoding is accomplished by converting an L to a K, inserting () at the appropriate indices, and then delegating to the original Codec[K]. I'm having trouble implementing the L => K conversion though.
def dropUnits[K <: HList, L <: HList](codec: Codec[K])(
implicit fltr: FilterNot.Aux[K, Unit, L]): Codec[L] = new Codec[L] {
override def decode(buffer: BitVector) =
codec.decode(buffer).map { case (rest, l) => (rest, l.filterNot[Unit]) }
override def encode(xs: L) = {
???
}
}
I've tried a few different solutions without luck. Is this possible with shapeless?
Source: (StackOverflow)
On version:
"org.typelevel" %% "scodec-core" % "1.5.0"
I'm trying to use coproduct functionality, as shown in the test case: "demonstrate fixing the codec to a known subtype" in CoproductsExample.scala
I keep getting the error: "could not find implicit value for parameter auto: scodec.codecs.CoproductBuilderAuto[my.class.here]"
I even copy pasted the example and could not get it to work:
import scalaz.\/
import shapeless._
import scodec.bits._
import scodec.codecs._
import scodec._
sealed trait Sprocket
object Sprocket {
implicit val discriminated: Discriminated[Sprocket, Int] = Discriminated(uint8)
}
def codec(d: Int): Codec[Sprocket] = Codec.coproduct[Sprocket].discriminatedBy(provide(d)).auto
I'll continue to look into this on my end, but was wondering if there was an issue fixed around this lately. I cloned the repo, and it worked from the clone - but not when I use the released version.
Source: (StackOverflow)